-- Logs begin at Sun 2022-10-09 00:10:45 UTC, end at Sun 2022-10-09 01:15:43 UTC. -- Oct 09 00:41:22.109336 dv-cinder-master-3-iscsi systemd[1]: Starting Devstack devstack@keystone.service... Oct 09 00:41:22.117855 dv-cinder-master-3-iscsi devstack@keystone.service[81441]: [uWSGI] getting INI configuration from /etc/keystone/keystone-uwsgi-public.ini Oct 09 00:41:22.119934 dv-cinder-master-3-iscsi devstack@keystone.service[81441]: *** Starting uWSGI 2.0.18-debian (64bit) on [Sun Oct 9 00:41:22 2022] *** Oct 09 00:41:22.119934 dv-cinder-master-3-iscsi devstack@keystone.service[81441]: compiled with version: 10.0.1 20200405 (experimental) [master revision 0be9efad938:fcb98e4978a:705510a708d3642c9c962beb663c476167e4e8a4] on 11 April 2020 11:15:55 Oct 09 00:41:22.119934 dv-cinder-master-3-iscsi devstack@keystone.service[81441]: os: Linux-5.4.0-72-generic #80-Ubuntu SMP Mon Apr 12 17:35:00 UTC 2021 Oct 09 00:41:22.119934 dv-cinder-master-3-iscsi devstack@keystone.service[81441]: nodename: dv-cinder-master-3-iscsi Oct 09 00:41:22.119934 dv-cinder-master-3-iscsi devstack@keystone.service[81441]: machine: x86_64 Oct 09 00:41:22.119934 dv-cinder-master-3-iscsi devstack@keystone.service[81441]: clock source: unix Oct 09 00:41:22.119934 dv-cinder-master-3-iscsi devstack@keystone.service[81441]: pcre jit disabled Oct 09 00:41:22.119934 dv-cinder-master-3-iscsi devstack@keystone.service[81441]: detected number of CPU cores: 4 Oct 09 00:41:22.119934 dv-cinder-master-3-iscsi devstack@keystone.service[81441]: current working directory: / Oct 09 00:41:22.119934 dv-cinder-master-3-iscsi devstack@keystone.service[81441]: detected binary path: /usr/bin/uwsgi-core Oct 09 00:41:22.119934 dv-cinder-master-3-iscsi devstack@keystone.service[81441]: your processes number limit is 47680 Oct 09 00:41:22.120500 dv-cinder-master-3-iscsi devstack@keystone.service[81441]: your memory page size is 4096 bytes Oct 09 00:41:22.120500 dv-cinder-master-3-iscsi devstack@keystone.service[81441]: detected max file descriptor number: 2048 Oct 09 00:41:22.120500 dv-cinder-master-3-iscsi devstack@keystone.service[81441]: lock engine: pthread robust mutexes Oct 09 00:41:22.120500 dv-cinder-master-3-iscsi devstack@keystone.service[81441]: thunder lock: enabled Oct 09 00:41:22.120500 dv-cinder-master-3-iscsi devstack@keystone.service[81441]: uwsgi socket 0 bound to UNIX address /var/run/uwsgi/keystone-wsgi-public.socket fd 4 Oct 09 00:41:22.120500 dv-cinder-master-3-iscsi devstack@keystone.service[81441]: Python version: 3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0] Oct 09 00:41:22.159240 dv-cinder-master-3-iscsi devstack@keystone.service[81441]: Python main interpreter initialized at 0x560dd1a659c0 Oct 09 00:41:22.159240 dv-cinder-master-3-iscsi devstack@keystone.service[81441]: python threads support enabled Oct 09 00:41:22.159240 dv-cinder-master-3-iscsi devstack@keystone.service[81441]: your server socket listen backlog is limited to 100 connections Oct 09 00:41:22.159240 dv-cinder-master-3-iscsi devstack@keystone.service[81441]: your mercy for graceful operations on workers is 90 seconds Oct 09 00:41:22.159564 dv-cinder-master-3-iscsi devstack@keystone.service[81441]: mapped 403077 bytes (393 KB) for 2 cores Oct 09 00:41:22.159678 dv-cinder-master-3-iscsi devstack@keystone.service[81441]: *** Operational MODE: preforking *** Oct 09 00:41:22.159937 dv-cinder-master-3-iscsi devstack@keystone.service[81441]: *** uWSGI is running in multiple interpreter mode *** Oct 09 00:41:22.159937 dv-cinder-master-3-iscsi devstack@keystone.service[81441]: spawned uWSGI master process (pid: 81441) Oct 09 00:41:22.160122 dv-cinder-master-3-iscsi systemd[1]: Started Devstack devstack@keystone.service. Oct 09 00:41:22.160645 dv-cinder-master-3-iscsi devstack@keystone.service[81441]: spawned uWSGI worker 1 (pid: 81452, cores: 1) Oct 09 00:41:22.160971 dv-cinder-master-3-iscsi devstack@keystone.service[81441]: spawned uWSGI worker 2 (pid: 81453, cores: 1) Oct 09 00:41:22.163691 dv-cinder-master-3-iscsi devstack@keystone.service[81441]: running "unix_signal:15 gracefully_kill_them_all" (master-start)... Oct 09 00:41:26.460171 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 09 00:41:26.460171 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: from cryptography.utils import int_from_bytes Oct 09 00:41:26.460954 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 09 00:41:26.460954 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: from cryptography.utils import int_from_bytes Oct 09 00:41:26.461402 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 09 00:41:26.461402 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: from cryptography.utils import int_from_bytes Oct 09 00:41:26.461504 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 09 00:41:26.461504 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: from cryptography.utils import int_from_bytes Oct 09 00:41:26.510183 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Oct 09 00:41:26.510344 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Oct 09 00:41:26.510437 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Oct 09 00:41:26.510527 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Oct 09 00:41:26.510601 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] command line args: [] {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Oct 09 00:41:26.510677 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] command line args: [] {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Oct 09 00:41:26.510806 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Oct 09 00:41:26.510894 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Oct 09 00:41:26.510973 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ================================================================================ {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Oct 09 00:41:26.511050 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] ================================================================================ {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Oct 09 00:41:26.511217 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] admin_token = **** {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.511305 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] admin_token = **** {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.511457 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] config_dir = [] {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.511543 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] config_dir = [] {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.511662 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.511753 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.511827 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] config_source = [] {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.511902 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] config_source = [] {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.512010 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] debug = True {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.512140 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] debug = True {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.512223 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.512504 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.512652 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.512751 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.512847 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.512950 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.513051 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.513154 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.513261 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.513364 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.513476 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] list_limit = None {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.513572 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] list_limit = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.513672 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] log_config_append = None {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.513768 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] log_config_append = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.513869 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.513979 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.514081 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] log_dir = None {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.514178 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] log_dir = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.514178 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] log_file = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.514371 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] log_file = None {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.514484 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.514563 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.514681 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.514804 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.515419 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.515496 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.515576 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.515733 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.515818 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.515899 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.516024 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.516109 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.516184 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.516316 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.516392 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.516515 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.516600 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.516681 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.516797 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.516879 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.517001 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.517082 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.517235 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.517327 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.517477 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.517566 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.517714 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.517799 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.517987 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.518077 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.518474 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.33/identity {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.518621 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.33/identity {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.518707 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] publish_errors = False {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.518779 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] publish_errors = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.518901 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.518980 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.519095 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.519181 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.519258 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.519374 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.519471 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.519572 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.519724 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.519812 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.519952 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] standard_threads = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.520039 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] standard_threads = False {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.520178 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.520271 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.520413 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.520492 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.520570 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.520642 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.520745 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] use_journal = False {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.520874 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] use_journal = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.520950 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] use_json = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.521025 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] use_json = False {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.521104 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] use_stderr = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.521176 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] use_stderr = False {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.521299 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] use_syslog = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.521383 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] use_syslog = False {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.521462 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.521539 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 09 00:41:26.521678 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.521765 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.521910 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.521997 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.522162 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.522269 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.522395 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.522474 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.522638 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.522724 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.522869 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.522947 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.523113 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.523192 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.523342 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.523428 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.523584 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.523692 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.523850 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.523930 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.524094 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.524179 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.524322 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.524399 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.524563 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.524650 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.524794 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.524886 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.525034 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.525113 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.525276 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.525355 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.525507 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.525593 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.525753 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.525832 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.526002 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.526089 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.526238 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.526313 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.526456 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.526542 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.526703 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.526781 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.526957 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.527036 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.527203 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.527295 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.527447 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.527525 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.527698 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.527780 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.527929 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.528010 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.528159 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.528236 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.528389 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.528468 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.528628 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.528713 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.528871 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.528950 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.529095 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.529174 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.529322 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.529399 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.529562 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.529640 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.529806 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.529885 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.530036 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.530120 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.530279 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.530357 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.530529 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.530615 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.530762 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.530841 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.530993 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.531071 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.531217 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.531296 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.531444 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.531528 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.531691 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.531776 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.531926 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.532008 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.532140 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.532221 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.532387 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.532519 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.532660 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.532787 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.532918 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.533047 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.533172 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.533298 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.533431 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.533555 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.533680 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.533763 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.533836 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.533912 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.534024 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.534124 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.534221 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.534317 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.534420 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.534517 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.534626 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.534730 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.534827 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.534984 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.535072 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.535149 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.535217 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.535288 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.535435 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.535617 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.535699 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.535844 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.535964 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.536040 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.536156 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.536241 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.536369 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.536514 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.536598 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.536685 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.536816 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.536943 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.537052 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.537132 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.537271 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.537399 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.537534 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.537655 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.537768 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.537847 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.538011 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.538097 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.538255 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.538343 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.538487 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.538606 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.538718 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.538806 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.538946 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.539025 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.539187 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.539269 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.539417 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.539506 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.539670 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.539753 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.539909 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.539999 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.540140 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.540233 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.540379 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.540460 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.540609 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.540690 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.540835 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.540923 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.541064 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.541142 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.541289 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.541368 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.541519 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.541610 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.541773 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.541852 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.542008 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.542087 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.542249 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.542590 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.542741 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.542822 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.542970 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.543050 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.543208 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.543369 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.543446 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.543517 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.543680 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.543763 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.543911 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.544003 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.544133 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.544214 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.544358 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.544437 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.544582 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.544661 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.544811 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.544892 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.545034 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.545114 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.545268 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.545342 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.545497 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.545577 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.545738 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.545817 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.545966 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.546045 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.546237 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.546327 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.546492 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.546580 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.546728 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.546835 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.546953 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] auth.external = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.547034 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] auth.external = None {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.547180 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.547262 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.547471 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.547555 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.547722 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.547804 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.547948 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] auth.password = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.548030 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] auth.password = None {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.548173 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] auth.token = None {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.548252 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] auth.token = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.548404 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.548483 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.548634 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.548717 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.548875 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.548954 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.549097 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.549176 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.549339 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.549421 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.549579 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.549658 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.549803 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.549885 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.550032 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] credential.caching = True {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.550119 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] credential.caching = True {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.550270 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.550349 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.550512 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.550603 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.550751 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.550837 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.550984 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.551064 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.551219 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.551298 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.551447 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.551528 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.551710 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.551793 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.551958 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.552038 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.552188 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.552270 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.552433 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.552513 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.552708 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.552788 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.552947 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.553029 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.553200 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.553289 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.553438 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.553519 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.553686 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.553767 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.553913 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] federation.caching = True {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.553992 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] federation.caching = True {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.554144 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.554223 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.554381 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.554462 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.554620 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.554699 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.554842 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.554930 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.555082 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.555165 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.555313 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.555392 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.555557 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.555667 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.555812 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.555899 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.556059 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.556141 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.556289 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.556377 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.556528 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.556608 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.556757 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] identity.caching = True {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.556836 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] identity.caching = True {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.556995 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.557144 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.557222 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.557295 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.557453 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.557535 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.557683 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.557771 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.557921 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.558001 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.558144 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.558224 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.558374 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.558463 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.558614 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.558694 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.558855 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.558935 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.559080 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.559168 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.559303 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.559383 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.559529 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.559631 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.559794 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.559886 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.560037 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.560118 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.560278 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.560360 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.560523 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.560617 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.560764 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.560844 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.561024 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.561151 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.561255 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.561397 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.561479 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.561552 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.562219 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.562302 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.562453 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.562543 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.562680 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.562768 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.562914 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.563020 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.563149 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.563241 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.563382 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.563463 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.563643 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.563726 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.563870 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.563964 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.564112 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.564193 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.564354 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.564433 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.564583 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.564664 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.564824 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.564910 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.565062 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.565142 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.565288 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.565368 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.565518 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.565598 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.565743 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.565822 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.565973 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.566054 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.566204 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.566283 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.566438 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.566517 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.566669 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.566750 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.567004 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.567083 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.567252 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.567331 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.567494 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.567575 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.567753 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.567841 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.567978 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.568058 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.568223 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.568305 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.568465 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.568548 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.568700 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.568788 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.568931 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.569012 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.569164 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.569244 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.569391 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.user = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.569469 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] ldap.user = None {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.569626 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.569707 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.569878 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.569957 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.570105 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.570184 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.570344 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.570434 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.570586 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.570670 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.570824 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.570903 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.571055 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.571136 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.571280 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.571360 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.571510 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.571610 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.571764 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.571847 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.571997 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.572077 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.572221 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.572300 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.572462 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.572542 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.572700 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.572779 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.572942 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.573030 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.573179 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.573260 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.573416 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.573495 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.573639 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.573718 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.573877 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.573958 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.574108 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.574188 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.574337 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.574418 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.574566 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.574653 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.574792 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.574871 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.575029 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.575108 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.575269 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.575350 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.575500 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.575579 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.575771 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.575851 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.575997 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.576079 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.576233 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.576312 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.576461 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.576541 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.576690 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.576770 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.576918 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.576997 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.577155 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.577236 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.577978 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.578065 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.578215 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.578296 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.578451 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.578540 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.578694 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] resource.caching = True {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.578780 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] resource.caching = True {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.578940 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.579019 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.579183 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.579263 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.579415 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.579495 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.579678 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.579763 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.579927 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.580006 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.580160 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.580242 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.580398 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.580477 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.580627 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.580781 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.580858 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.580929 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.581076 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] role.caching = True {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.581156 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] role.caching = True {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.581315 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] role.driver = sql {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.581402 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] role.driver = sql {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.581537 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.581617 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.581780 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.581862 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.582019 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.582112 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.582261 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.582340 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.582500 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.582579 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.582741 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.582828 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.582978 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.583111 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.583252 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.583381 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.583508 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.583659 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.583785 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.583913 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.584038 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.584163 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.584290 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.584426 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.584553 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.584680 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.584805 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.584933 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.585075 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.585212 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.585339 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.585466 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.585466 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.585705 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.585833 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.585967 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.586094 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.586220 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.586347 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.586482 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.586622 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.586764 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.586893 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.587019 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.587145 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.587272 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.587406 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.587539 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.587690 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.587819 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.587946 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.588074 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.588211 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.588338 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.588477 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.588605 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.588731 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.588858 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.588991 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.589118 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.589244 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.589371 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.589495 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.589620 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.589760 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.589887 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.590014 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] token.caching = True {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.590141 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] token.caching = True {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.590279 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.590406 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.590541 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.590667 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.590796 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.590927 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.591052 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.591180 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.591313 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.591440 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.591565 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.591709 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.591839 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.591977 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.592115 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.592249 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.592375 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.592500 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.592627 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.592752 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.592888 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.593015 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.593140 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.593268 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.593395 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.593522 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.593657 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.593803 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.593933 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.594068 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.594208 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.594344 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.594472 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.594606 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.594733 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] database.connection = **** {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.594866 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] database.connection = **** {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.594993 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.595119 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.595246 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.595372 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.595498 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.595665 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.595790 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.595922 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.596047 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.596170 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.596292 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.596423 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.596547 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.596671 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.596794 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.596916 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.597040 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.597171 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.597295 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.597429 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.597553 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.597676 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.597799 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.597930 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.598055 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.598178 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.598302 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.598425 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.598548 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.598680 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.598803 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.598926 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.599059 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.599198 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.599323 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.599459 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 09 00:41:26.599583 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Oct 09 00:41:26.599766 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81453) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Oct 09 00:41:26.627225 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 09 00:41:26.627438 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 09 00:41:26.627583 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81453) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 09 00:41:26.627735 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 09 00:41:26.627870 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81453) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 09 00:41:26.628056 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81453) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 09 00:41:26.635888 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 09 00:41:26.636332 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81453) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 09 00:41:26.663947 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 09 00:41:26.664172 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 09 00:41:26.664451 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81453) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 09 00:41:26.664682 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81453) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 09 00:41:26.717660 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81453) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 09 00:41:26.718617 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 09 00:41:26.766441 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81453) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 09 00:41:26.766670 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81453) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 09 00:41:26.766819 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 09 00:41:26.766999 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 09 00:41:26.921989 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81453) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 09 00:41:26.922257 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81453) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 09 00:41:26.922421 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81453) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 09 00:41:26.922558 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 09 00:41:26.922691 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81453) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 09 00:41:26.922841 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 09 00:41:26.922972 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81453) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 09 00:41:26.923102 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 09 00:41:26.923230 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81453) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 09 00:41:26.923371 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 09 00:41:26.923371 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 09 00:41:26.923658 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 09 00:41:26.946163 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81453) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 09 00:41:26.946321 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 09 00:41:26.946453 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81453) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 09 00:41:26.946592 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 09 00:41:26.946730 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81453) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 09 00:41:26.946859 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 09 00:41:26.946986 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81453) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 09 00:41:26.947126 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 09 00:41:26.947255 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81453) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 09 00:41:26.947395 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 09 00:41:26.947523 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81453) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 09 00:41:26.947678 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 09 00:41:26.947813 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81453) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 09 00:41:26.947946 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 09 00:41:26.958486 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81453) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 09 00:41:26.958660 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 09 00:41:26.971098 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81453) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 09 00:41:26.971265 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81453) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 09 00:41:26.971407 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 09 00:41:26.971546 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81453) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 09 00:41:26.971699 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 09 00:41:26.971827 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81453) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 09 00:41:26.971956 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 09 00:41:26.972084 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81453) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 09 00:41:26.972210 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 09 00:41:26.972347 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81453) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 09 00:41:26.972475 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 09 00:41:26.972602 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81453) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 09 00:41:26.972739 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 09 00:41:26.972739 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 09 00:41:26.989485 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:26.989693 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 09 00:41:26.989968 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:26.990098 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:26.990228 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 09 00:41:26.990356 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 09 00:41:26.990496 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:26.990624 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:26.990751 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 09 00:41:26.990878 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:26.991015 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:26.991151 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:26.991281 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:26.991408 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:26.991534 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:26.991695 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.001363 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.001639 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.001770 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.001897 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.002024 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.002162 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.002303 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.002552 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.009165 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81453) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 09 00:41:27.009458 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81452) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 09 00:41:27.012077 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81453) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 09 00:41:27.012362 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.012630 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81452) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 09 00:41:27.012713 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.012909 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.012990 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.013187 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.013278 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.013445 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.013528 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.013714 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.013798 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.013978 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.014062 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.014243 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.014324 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.014512 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.014602 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.014811 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.014924 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.015082 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.015347 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.037844 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.038193 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.039301 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81453) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 09 00:41:27.039583 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.039689 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81452) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 09 00:41:27.039935 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.044585 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81453) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 09 00:41:27.044862 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81452) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 09 00:41:27.044947 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.045138 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.045220 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.045412 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.052042 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81453) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 09 00:41:27.052300 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81452) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 09 00:41:27.052385 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.052583 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.057908 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81453) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 09 00:41:27.058104 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81452) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 09 00:41:27.058189 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.058393 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.058477 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.058674 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.058767 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.058956 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.059044 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.059231 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.059314 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.059504 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.059609 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.059798 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.059881 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.060065 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.076647 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.076997 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.077105 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.077227 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.079477 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81453) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 09 00:41:27.079581 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81452) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 09 00:41:27.082721 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.082813 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.083016 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.083101 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.087224 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81453) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 09 00:41:27.087317 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81452) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 09 00:41:27.090355 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81453) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 09 00:41:27.090444 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81452) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 09 00:41:27.094125 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.094232 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.094422 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.094508 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.094702 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.094785 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.094983 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.095067 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.095259 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.095342 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.095537 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.095673 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.111813 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81453) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 09 00:41:27.111936 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81452) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 09 00:41:27.112103 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.112230 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.112374 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.112495 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.112641 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.112754 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.119416 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.119507 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.120978 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.121115 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.122495 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.122581 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.122772 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.122857 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.125922 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81452) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 09 00:41:27.126014 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81453) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 09 00:41:27.126204 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.126289 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.126487 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.126584 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.126770 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.126855 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.127167 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.127295 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.138137 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81452) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 09 00:41:27.138234 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81453) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 09 00:41:27.138416 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.138501 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.138693 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.138790 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.138980 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.139065 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.139356 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.139441 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.139747 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.140334 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.140532 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.140616 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.155713 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81452) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 09 00:41:27.155852 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81453) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 09 00:41:27.158584 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81452) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 09 00:41:27.158748 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81453) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 09 00:41:27.161446 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.161634 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.162985 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.163113 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.164467 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81452) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 09 00:41:27.164608 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81453) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 09 00:41:27.164750 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.164893 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.165155 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.165301 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.171714 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.171928 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.173155 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81452) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 09 00:41:27.173389 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81453) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 09 00:41:27.176042 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.176291 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.176440 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.176687 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.176773 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.176974 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.177060 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.177255 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.184664 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81452) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 09 00:41:27.184951 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.185048 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81453) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 09 00:41:27.185231 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.185323 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.185567 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.192551 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81452) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 09 00:41:27.192833 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.192917 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81453) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 09 00:41:27.193111 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.193195 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.193381 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.193465 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.193650 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.193736 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.193915 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.194000 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.194185 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.194277 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.194470 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.194564 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.194743 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.194837 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.195016 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.195099 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.195283 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.195365 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.195549 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.195651 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.195836 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.195920 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.196101 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.196239 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.196522 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81453) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 09 00:41:27.230320 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x560dd1a659c0 pid: 81452 (default app) Oct 09 00:41:27.230548 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x560dd1a659c0 pid: 81453 (default app) Oct 09 00:41:27.233549 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65cc5137-c714-4569-858c-9f6a1014b35e None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:27.233822 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65cc5137-c714-4569-858c-9f6a1014b35e None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:27.234094 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65cc5137-c714-4569-858c-9f6a1014b35e None None] PATH_INFO: `/v3` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:27.238192 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1/1] 10.222.0.33 () {50 vars in 791 bytes} [Sun Oct 9 00:41:27 2022] GET /identity/v3/ => generated 255 bytes in 7 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:41:33.953223 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a0f6de-3fa9-446c-8d1c-30b3e0e025e6 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:33.953739 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a0f6de-3fa9-446c-8d1c-30b3e0e025e6 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:33.953739 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a0f6de-3fa9-446c-8d1c-30b3e0e025e6 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:33.957990 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1/2] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:41:33 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 09 00:41:33.963299 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bfc2db7-4f4a-4521-8774-443f97316f42 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:33.963747 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bfc2db7-4f4a-4521-8774-443f97316f42 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:33.963882 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bfc2db7-4f4a-4521-8774-443f97316f42 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:34.033778 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: INFO dbcounter [None req-9bfc2db7-4f4a-4521-8774-443f97316f42 None None] Registered counter for database keystone Oct 09 00:41:34.053058 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG oslo_db.sqlalchemy.engines [None req-9bfc2db7-4f4a-4521-8774-443f97316f42 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81452) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Oct 09 00:41:34.059187 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writer thread running {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Oct 09 00:41:34.235261 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG passlib.handlers.bcrypt [None req-9bfc2db7-4f4a-4521-8774-443f97316f42 None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81452) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Oct 09 00:41:34.235567 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG passlib.handlers.bcrypt [None req-9bfc2db7-4f4a-4521-8774-443f97316f42 None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81452) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Oct 09 00:41:34.294052 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-9bfc2db7-4f4a-4521-8774-443f97316f42 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:41:34.296193 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-9bfc2db7-4f4a-4521-8774-443f97316f42 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:41:34.421472 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 2/3] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:41:33 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 459 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Oct 09 00:41:34.876235 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6a0f6de-3fa9-446c-8d1c-30b3e0e025e6 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:41:34.945099 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: INFO dbcounter [None req-c6a0f6de-3fa9-446c-8d1c-30b3e0e025e6 None None] Registered counter for database keystone Oct 09 00:41:34.966378 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG oslo_db.sqlalchemy.engines [None req-c6a0f6de-3fa9-446c-8d1c-30b3e0e025e6 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81453) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Oct 09 00:41:34.973211 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writer thread running {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Oct 09 00:41:35.090003 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6a0f6de-3fa9-446c-8d1c-30b3e0e025e6 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:41:35.095010 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65f58d1f-aa20-460c-8f89-a11a32e3bcd7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:41:35.095659 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65f58d1f-aa20-460c-8f89-a11a32e3bcd7 admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:35.095949 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65f58d1f-aa20-460c-8f89-a11a32e3bcd7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:35.096220 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65f58d1f-aa20-460c-8f89-a11a32e3bcd7 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:35.104414 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65f58d1f-aa20-460c-8f89-a11a32e3bcd7 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:41:35.155531 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: WARNING keystone.common.rbac_enforcer.enforcer [None req-65f58d1f-aa20-460c-8f89-a11a32e3bcd7 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Oct 09 00:41:35.155839 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65f58d1f-aa20-460c-8f89-a11a32e3bcd7 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:41:35.165044 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 2/4] 10.222.0.33 () {58 vars in 1324 bytes} [Sun Oct 9 00:41:34 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 289 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Oct 09 00:41:35.171312 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bfc2db7-4f4a-4521-8774-443f97316f42 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:41:35.188042 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bfc2db7-4f4a-4521-8774-443f97316f42 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:41:35.192739 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9da87cbb-3417-44d5-9c9c-a0a2f9484a1a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:41:35.193831 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9da87cbb-3417-44d5-9c9c-a0a2f9484a1a admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:35.194115 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9da87cbb-3417-44d5-9c9c-a0a2f9484a1a admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:35.194391 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9da87cbb-3417-44d5-9c9c-a0a2f9484a1a admin admin] PATH_INFO: `/v3/projects/34e0fbee16724df3abcac7131a4822aa` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:35.196007 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9da87cbb-3417-44d5-9c9c-a0a2f9484a1a admin admin] RBAC: Authorizing `identity:get_project(project_id=34e0fbee16724df3abcac7131a4822aa)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:41:35.248213 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: WARNING keystone.common.rbac_enforcer.enforcer [None req-9da87cbb-3417-44d5-9c9c-a0a2f9484a1a admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Oct 09 00:41:35.248567 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9da87cbb-3417-44d5-9c9c-a0a2f9484a1a admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:41:35.253028 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 3/5] 10.222.0.33 () {56 vars in 1207 bytes} [Sun Oct 9 00:41:35 2022] GET /identity/v3/projects/34e0fbee16724df3abcac7131a4822aa => generated 344 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:41:37.377054 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc5efaab-78dd-48d6-8338-89d142a325fd None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:37.377581 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc5efaab-78dd-48d6-8338-89d142a325fd None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:37.377581 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc5efaab-78dd-48d6-8338-89d142a325fd None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:37.381661 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 3/6] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:41:37 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:41:37.387077 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0112e915-a794-42d9-ba9d-ddea9ad69fe3 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:37.387462 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0112e915-a794-42d9-ba9d-ddea9ad69fe3 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:37.387668 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0112e915-a794-42d9-ba9d-ddea9ad69fe3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:37.408397 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-0112e915-a794-42d9-ba9d-ddea9ad69fe3 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:41:37.409898 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-0112e915-a794-42d9-ba9d-ddea9ad69fe3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:41:37.418206 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 4/7] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:41:37 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Oct 09 00:41:37.695431 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc5efaab-78dd-48d6-8338-89d142a325fd None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:41:37.708062 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc5efaab-78dd-48d6-8338-89d142a325fd None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:41:37.712241 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-571ad8f0-6fd8-4c6b-9968-46c3335b3318 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:41:37.712830 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-571ad8f0-6fd8-4c6b-9968-46c3335b3318 admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:37.713111 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-571ad8f0-6fd8-4c6b-9968-46c3335b3318 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:37.713382 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-571ad8f0-6fd8-4c6b-9968-46c3335b3318 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:37.721179 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-571ad8f0-6fd8-4c6b-9968-46c3335b3318 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:41:37.722933 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-571ad8f0-6fd8-4c6b-9968-46c3335b3318 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:41:37.731433 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 4/8] 10.222.0.33 () {58 vars in 1324 bytes} [Sun Oct 9 00:41:37 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 09 00:41:37.736753 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0112e915-a794-42d9-ba9d-ddea9ad69fe3 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:41:37.745722 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0112e915-a794-42d9-ba9d-ddea9ad69fe3 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:41:37.750611 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d393c6f-d74d-4079-9ccf-b3e1c5ddc541 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:41:37.751232 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d393c6f-d74d-4079-9ccf-b3e1c5ddc541 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:37.751514 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d393c6f-d74d-4079-9ccf-b3e1c5ddc541 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:37.751826 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d393c6f-d74d-4079-9ccf-b3e1c5ddc541 admin admin] PATH_INFO: `/v3/users/d9c8218a78034540ba9ed09b697588a8` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:37.753128 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d393c6f-d74d-4079-9ccf-b3e1c5ddc541 admin admin] RBAC: Authorizing `identity:get_user(user_id=d9c8218a78034540ba9ed09b697588a8)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:41:37.754948 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d393c6f-d74d-4079-9ccf-b3e1c5ddc541 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:41:37.756489 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 5/9] 10.222.0.33 () {56 vars in 1198 bytes} [Sun Oct 9 00:41:37 2022] GET /identity/v3/users/d9c8218a78034540ba9ed09b697588a8 => generated 247 bytes in 20 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:41:39.953443 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6099b085-06ab-42e9-8ae8-7ee884353a7e None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:39.953979 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6099b085-06ab-42e9-8ae8-7ee884353a7e None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:39.953979 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6099b085-06ab-42e9-8ae8-7ee884353a7e None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:39.955295 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 5/10] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:41:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:41:39.960468 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fecf1a81-0321-4e21-bcb5-dfbd57e4c598 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:39.960854 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fecf1a81-0321-4e21-bcb5-dfbd57e4c598 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:39.961024 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fecf1a81-0321-4e21-bcb5-dfbd57e4c598 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:39.967618 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51dfab56-abb8-4ba5-9870-518115f131b6 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:39.967923 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51dfab56-abb8-4ba5-9870-518115f131b6 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:39.968197 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51dfab56-abb8-4ba5-9870-518115f131b6 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:39.969441 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 6/11] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:41:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 09 00:41:39.974747 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af82e0bf-8c96-49cf-b7da-2a37a876c232 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:39.974747 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af82e0bf-8c96-49cf-b7da-2a37a876c232 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:39.974960 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af82e0bf-8c96-49cf-b7da-2a37a876c232 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:39.979883 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-fecf1a81-0321-4e21-bcb5-dfbd57e4c598 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:41:39.981042 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-fecf1a81-0321-4e21-bcb5-dfbd57e4c598 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:41:40.016274 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG passlib.handlers.bcrypt [None req-af82e0bf-8c96-49cf-b7da-2a37a876c232 None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81453) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Oct 09 00:41:40.016655 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG passlib.handlers.bcrypt [None req-af82e0bf-8c96-49cf-b7da-2a37a876c232 None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81453) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Oct 09 00:41:40.062757 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 6/12] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:41:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 09 00:41:40.066700 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-af82e0bf-8c96-49cf-b7da-2a37a876c232 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:41:40.068321 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-af82e0bf-8c96-49cf-b7da-2a37a876c232 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:41:40.171323 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 7/13] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:41:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 198 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 09 00:41:40.331571 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fecf1a81-0321-4e21-bcb5-dfbd57e4c598 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:41:40.355879 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:41:40.441541 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fecf1a81-0321-4e21-bcb5-dfbd57e4c598 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:41:40.472967 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af82e0bf-8c96-49cf-b7da-2a37a876c232 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:41:40.474695 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-194ed7ef-206d-4293-b004-a79336d7679a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:41:40.475464 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194ed7ef-206d-4293-b004-a79336d7679a None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:40.475780 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194ed7ef-206d-4293-b004-a79336d7679a None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:40.476069 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194ed7ef-206d-4293-b004-a79336d7679a None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:40.487754 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-194ed7ef-206d-4293-b004-a79336d7679a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:41:40.496898 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-194ed7ef-206d-4293-b004-a79336d7679a None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 09 00:41:40.496898 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:41:40.496898 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:41:40.496898 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:41:40.496898 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:41:40.496898 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:41:40.496898 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:41:40.496898 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:41:40.496898 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:41:40.496898 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:41:40.496898 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:41:40.496898 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:41:40.496898 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:41:40.496898 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:41:40.496898 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:41:40.496898 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:41:40.498471 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:41:40.498471 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:41:40.498471 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:41:40.498471 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:41:40.498471 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:41:40.498471 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:41:40.498471 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:41:40.498471 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:41:40.498471 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:41:40.498471 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:41:40.498471 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:41:40.498471 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:41:40.498471 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:41:40.498471 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:41:40.498471 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:41:40.498471 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:41:40.499287 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:41:40.499287 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:41:40.499287 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:41:40.499287 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:41:40.499287 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:41:40.499287 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:41:40.499287 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:41:40.499287 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:41:40.499287 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:41:40.499287 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:41:40.499287 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 09 00:41:40.499287 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:41:40.499287 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 7/14] 10.222.0.33 () {56 vars in 1096 bytes} [Sun Oct 9 00:41:40 2022] GET /identity/v3/roles/admin => generated 83 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:41:40.504294 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-194ed7ef-206d-4293-b004-a79336d7679a None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:41:40.587202 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af82e0bf-8c96-49cf-b7da-2a37a876c232 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:41:40.589434 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:41:40.590826 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:41:40.613119 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-194ed7ef-206d-4293-b004-a79336d7679a None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:41:40.631753 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2d3527a-dfba-49c7-9db8-76dcb6464e1f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:41:40.632613 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d3527a-dfba-49c7-9db8-76dcb6464e1f None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:40.632894 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d3527a-dfba-49c7-9db8-76dcb6464e1f None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:40.633173 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d3527a-dfba-49c7-9db8-76dcb6464e1f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:40.649685 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f6b477-f8c3-493c-98b7-279f6855503d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:41:40.650287 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f6b477-f8c3-493c-98b7-279f6855503d None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:40.650630 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f6b477-f8c3-493c-98b7-279f6855503d None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:40.650906 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f6b477-f8c3-493c-98b7-279f6855503d None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:40.651575 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5f6b477-f8c3-493c-98b7-279f6855503d None admin] RBAC: Adding query filter params (name=admin) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:41:40.651901 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5f6b477-f8c3-493c-98b7-279f6855503d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:41:40.653287 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5f6b477-f8c3-493c-98b7-279f6855503d None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:41:40.666702 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 8/15] 10.222.0.33 () {56 vars in 1099 bytes} [Sun Oct 9 00:41:40 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:41:40.672953 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f6b477-f8c3-493c-98b7-279f6855503d None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:41:40.708513 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2d3527a-dfba-49c7-9db8-76dcb6464e1f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:41:40.710052 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2d3527a-dfba-49c7-9db8-76dcb6464e1f None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:41:40.782903 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f6b477-f8c3-493c-98b7-279f6855503d None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:41:40.810961 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 8/16] 10.222.0.33 () {58 vars in 1282 bytes} [Sun Oct 9 00:41:40 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 339 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 09 00:41:40.816440 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2d3527a-dfba-49c7-9db8-76dcb6464e1f None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:41:40.817019 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a4d7196-fa25-42b8-bef2-321b40cfa242 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:41:40.817655 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a4d7196-fa25-42b8-bef2-321b40cfa242 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:40.817938 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a4d7196-fa25-42b8-bef2-321b40cfa242 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:40.818217 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a4d7196-fa25-42b8-bef2-321b40cfa242 None admin] PATH_INFO: `/v3/users/d9c8218a78034540ba9ed09b697588a8` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:40.819064 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:41:40.819348 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a4d7196-fa25-42b8-bef2-321b40cfa242 None admin] RBAC: Authorizing `identity:get_user(user_id=d9c8218a78034540ba9ed09b697588a8)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:41:40.821094 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a4d7196-fa25-42b8-bef2-321b40cfa242 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:41:40.822704 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 9/17] 10.222.0.33 () {56 vars in 1177 bytes} [Sun Oct 9 00:41:40 2022] GET /identity/v3/users/d9c8218a78034540ba9ed09b697588a8 => generated 247 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:41:40.828125 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a4d7196-fa25-42b8-bef2-321b40cfa242 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:41:40.842822 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:41:40.911601 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2d3527a-dfba-49c7-9db8-76dcb6464e1f None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:41:40.926777 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a4d7196-fa25-42b8-bef2-321b40cfa242 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:41:40.945906 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24524385-cb7c-4c25-94da-40ee1b744dd0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:41:40.946753 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24524385-cb7c-4c25-94da-40ee1b744dd0 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:40.947037 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24524385-cb7c-4c25-94da-40ee1b744dd0 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:40.947315 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24524385-cb7c-4c25-94da-40ee1b744dd0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:40.960948 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e5e60a0-d2c9-4b6b-b4c8-6487f4dab495 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:41:40.961584 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5e60a0-d2c9-4b6b-b4c8-6487f4dab495 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:40.961858 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5e60a0-d2c9-4b6b-b4c8-6487f4dab495 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:40.962126 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5e60a0-d2c9-4b6b-b4c8-6487f4dab495 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:40.963210 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e5e60a0-d2c9-4b6b-b4c8-6487f4dab495 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:41:40.964690 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e5e60a0-d2c9-4b6b-b4c8-6487f4dab495 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:41:40.966210 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24524385-cb7c-4c25-94da-40ee1b744dd0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:41:40.966621 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 10/18] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:41:40 2022] GET /identity/v3/domains/default => generated 199 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:41:40.967785 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24524385-cb7c-4c25-94da-40ee1b744dd0 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:41:40.971865 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e5e60a0-d2c9-4b6b-b4c8-6487f4dab495 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:41:40.988106 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-24524385-cb7c-4c25-94da-40ee1b744dd0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:41:40.988106 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:41:40.988106 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:41:40.988106 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:41:40.988106 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:41:40.988106 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:41:40.988106 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:41:40.988106 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:41:40.988106 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:41:40.988106 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:41:40.988106 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:41:40.988106 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:41:40.988106 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:41:40.988106 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:41:40.988106 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:41:40.988106 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:41:40.989093 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:41:40.989093 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:41:40.989093 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:41:40.989093 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:41:40.989093 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:41:40.989093 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:41:40.989093 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:41:40.989093 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:41:40.989093 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:41:40.989093 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:41:40.989093 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:41:40.989093 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:41:40.989093 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:41:40.989093 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:41:40.989093 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:41:40.989093 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:41:40.989093 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:41:40.989869 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:41:40.989869 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:41:40.989869 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:41:40.989869 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 9/19] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:41:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 09 00:41:40.994857 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24524385-cb7c-4c25-94da-40ee1b744dd0 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:41:41.051270 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:41:41.055679 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:41:41.076016 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e5e60a0-d2c9-4b6b-b4c8-6487f4dab495 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:41:41.099682 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24524385-cb7c-4c25-94da-40ee1b744dd0 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:41:41.111381 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df7af35e-3484-41e8-b422-012ed0e74eb3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:41:41.112016 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df7af35e-3484-41e8-b422-012ed0e74eb3 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:41.112297 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df7af35e-3484-41e8-b422-012ed0e74eb3 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:41.112570 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df7af35e-3484-41e8-b422-012ed0e74eb3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:41.113290 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df7af35e-3484-41e8-b422-012ed0e74eb3 None admin] RBAC: Adding query filter params (role.id=092036ee46ac427baf4d56468575f697, scope.domain.id=default, user.id=d9c8218a78034540ba9ed09b697588a8) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:41:41.113593 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df7af35e-3484-41e8-b422-012ed0e74eb3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:41:41.115037 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df7af35e-3484-41e8-b422-012ed0e74eb3 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:41:41.121773 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 11/20] 10.222.0.33 () {56 vars in 1322 bytes} [Sun Oct 9 00:41:40 2022] GET /identity/v3/role_assignments?user.id=d9c8218a78034540ba9ed09b697588a8&scope.domain.id=default&role.id=092036ee46ac427baf4d56468575f697 => generated 233 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:41:41.132996 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd86f38a-aef2-4e59-9a27-6bfe3eab9631 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:41:41.133648 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd86f38a-aef2-4e59-9a27-6bfe3eab9631 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:41.133927 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd86f38a-aef2-4e59-9a27-6bfe3eab9631 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:41.134194 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd86f38a-aef2-4e59-9a27-6bfe3eab9631 None admin] PATH_INFO: `/v3/domains` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:41.134849 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd86f38a-aef2-4e59-9a27-6bfe3eab9631 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:41:41.135150 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd86f38a-aef2-4e59-9a27-6bfe3eab9631 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:41:41.136607 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd86f38a-aef2-4e59-9a27-6bfe3eab9631 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:41:41.155883 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 10/21] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:41:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:41:43.585080 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc37336d-5ba2-47a9-b607-617137c6820c None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:43.586125 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc37336d-5ba2-47a9-b607-617137c6820c None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:43.586626 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc37336d-5ba2-47a9-b607-617137c6820c None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:43.588413 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 12/22] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:41:43 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:41:43.594258 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d3ce184-53bc-4f81-889b-bce85856d3e0 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:43.595075 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d3ce184-53bc-4f81-889b-bce85856d3e0 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:43.595558 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d3ce184-53bc-4f81-889b-bce85856d3e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:43.619188 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-1d3ce184-53bc-4f81-889b-bce85856d3e0 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:41:43.620729 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-1d3ce184-53bc-4f81-889b-bce85856d3e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:41:43.657554 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-156e5351-e814-4c1c-b102-8980311962e1 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:43.658242 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-156e5351-e814-4c1c-b102-8980311962e1 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:43.658770 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-156e5351-e814-4c1c-b102-8980311962e1 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:43.660300 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 13/23] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:41:43 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 09 00:41:43.674287 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8023253c-8fda-4258-a8b0-bff1957ac51f None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:43.674838 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8023253c-8fda-4258-a8b0-bff1957ac51f None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:43.675316 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8023253c-8fda-4258-a8b0-bff1957ac51f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:43.708743 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-8023253c-8fda-4258-a8b0-bff1957ac51f None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:41:43.710916 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-8023253c-8fda-4258-a8b0-bff1957ac51f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:41:43.765162 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 11/24] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:41:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 09 00:41:43.853958 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 14/25] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:41:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 09 00:41:44.268411 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3168d9eb-e589-4d9f-8f43-d102145b1faa None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:44.269085 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3168d9eb-e589-4d9f-8f43-d102145b1faa None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:44.269578 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3168d9eb-e589-4d9f-8f43-d102145b1faa None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:44.271111 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 12/26] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:41:44 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:41:44.275690 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da4eea76-244c-439a-99ef-d773afaa137b None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:44.276222 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da4eea76-244c-439a-99ef-d773afaa137b None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:44.276714 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da4eea76-244c-439a-99ef-d773afaa137b None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:44.278742 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 15/27] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:41:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:41:44.281228 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca1381d-c65e-4264-8919-58349f70719a None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:44.281733 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca1381d-c65e-4264-8919-58349f70719a None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:44.282205 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca1381d-c65e-4264-8919-58349f70719a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:44.283938 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a380b3-5944-4ccb-a4a3-bd270e4a05a5 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:44.284428 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a380b3-5944-4ccb-a4a3-bd270e4a05a5 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:44.284910 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a380b3-5944-4ccb-a4a3-bd270e4a05a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:44.317431 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-aca1381d-c65e-4264-8919-58349f70719a None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:41:44.318624 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-aca1381d-c65e-4264-8919-58349f70719a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:41:44.319121 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-c1a380b3-5944-4ccb-a4a3-bd270e4a05a5 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:41:44.320713 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-c1a380b3-5944-4ccb-a4a3-bd270e4a05a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:41:44.378606 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:41:44.406693 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 13/28] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:41:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 09 00:41:44.407535 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aca1381d-c65e-4264-8919-58349f70719a None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:41:44.416411 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 16/29] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:41:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 09 00:41:44.417072 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1a380b3-5944-4ccb-a4a3-bd270e4a05a5 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:41:44.433365 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:41:44.607938 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1a380b3-5944-4ccb-a4a3-bd270e4a05a5 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:41:44.617070 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aca1381d-c65e-4264-8919-58349f70719a None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:41:44.659477 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-872b504f-0322-4fba-ae01-03ccd3420256 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:41:44.660170 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-426f64a2-716c-496b-ac29-69653fe290fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:41:44.660742 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-872b504f-0322-4fba-ae01-03ccd3420256 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:44.660847 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-426f64a2-716c-496b-ac29-69653fe290fc None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:44.661073 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-426f64a2-716c-496b-ac29-69653fe290fc None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:44.661348 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-426f64a2-716c-496b-ac29-69653fe290fc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:44.661726 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-872b504f-0322-4fba-ae01-03ccd3420256 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:44.662308 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-872b504f-0322-4fba-ae01-03ccd3420256 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:44.675146 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-872b504f-0322-4fba-ae01-03ccd3420256 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:41:44.681423 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-426f64a2-716c-496b-ac29-69653fe290fc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:41:44.682475 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-872b504f-0322-4fba-ae01-03ccd3420256 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 09 00:41:44.682475 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:41:44.682475 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:41:44.682475 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:41:44.682475 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:41:44.682475 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:41:44.682475 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:41:44.682475 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:41:44.682475 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:41:44.682475 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:41:44.682475 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:41:44.682475 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:41:44.682475 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:41:44.682475 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:41:44.682475 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:41:44.682475 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:41:44.683271 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:41:44.683271 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:41:44.683271 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:41:44.683271 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:41:44.683271 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:41:44.683271 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:41:44.683271 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:41:44.683271 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:41:44.683271 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:41:44.683271 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:41:44.683271 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:41:44.683271 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:41:44.683271 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:41:44.683271 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:41:44.683271 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:41:44.683271 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:41:44.684039 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:41:44.684039 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:41:44.684039 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:41:44.684039 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:41:44.684039 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:41:44.684039 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:41:44.684039 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:41:44.684039 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:41:44.684039 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:41:44.684039 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:41:44.684039 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 09 00:41:44.684039 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:41:44.684636 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-426f64a2-716c-496b-ac29-69653fe290fc None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:41:44.686986 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 14/30] 10.222.0.33 () {56 vars in 1096 bytes} [Sun Oct 9 00:41:44 2022] GET /identity/v3/roles/admin => generated 83 bytes in 280 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:41:44.692531 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-872b504f-0322-4fba-ae01-03ccd3420256 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:41:44.702399 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-426f64a2-716c-496b-ac29-69653fe290fc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:41:44.702399 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:41:44.702399 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:41:44.702399 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:41:44.702399 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:41:44.702399 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:41:44.702399 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:41:44.702399 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:41:44.702399 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:41:44.702399 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:41:44.702399 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:41:44.702399 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:41:44.702399 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:41:44.702399 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:41:44.702399 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:41:44.702399 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:41:44.703244 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:41:44.703244 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:41:44.703244 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:41:44.703244 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:41:44.703244 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:41:44.703244 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:41:44.703244 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:41:44.703244 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:41:44.703244 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:41:44.703244 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:41:44.703244 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:41:44.703244 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:41:44.703244 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:41:44.703244 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:41:44.703244 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:41:44.703244 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:41:44.703244 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:41:44.704048 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:41:44.704048 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:41:44.704048 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:41:44.705885 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 17/31] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:41:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 289 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:41:44.732341 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-426f64a2-716c-496b-ac29-69653fe290fc None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:41:44.739273 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:41:44.824705 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-872b504f-0322-4fba-ae01-03ccd3420256 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:41:44.836510 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:41:44.868499 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d7c6360-4289-42a1-ab52-c43be88c7506 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:41:44.869036 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d7c6360-4289-42a1-ab52-c43be88c7506 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:44.869321 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d7c6360-4289-42a1-ab52-c43be88c7506 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:44.869599 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d7c6360-4289-42a1-ab52-c43be88c7506 None admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:44.870265 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d7c6360-4289-42a1-ab52-c43be88c7506 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:41:44.870569 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d7c6360-4289-42a1-ab52-c43be88c7506 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:41:44.872539 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d7c6360-4289-42a1-ab52-c43be88c7506 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:41:44.885520 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-426f64a2-716c-496b-ac29-69653fe290fc None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:41:44.892422 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 15/32] 10.222.0.33 () {56 vars in 1099 bytes} [Sun Oct 9 00:41:44 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:41:44.892422 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d7c6360-4289-42a1-ab52-c43be88c7506 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:41:44.925903 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7086ea3-1327-44d9-b271-3c5049f5a4c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:41:44.926851 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7086ea3-1327-44d9-b271-3c5049f5a4c2 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:44.927389 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7086ea3-1327-44d9-b271-3c5049f5a4c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:44.927910 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7086ea3-1327-44d9-b271-3c5049f5a4c2 None admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:44.928734 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7086ea3-1327-44d9-b271-3c5049f5a4c2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:41:44.929223 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7086ea3-1327-44d9-b271-3c5049f5a4c2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:41:44.930806 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7086ea3-1327-44d9-b271-3c5049f5a4c2 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:41:44.958929 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 18/33] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:41:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:41:44.963999 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7086ea3-1327-44d9-b271-3c5049f5a4c2 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:41:45.013748 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:41:45.019343 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d7c6360-4289-42a1-ab52-c43be88c7506 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:41:45.062447 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-553bf3a0-ecc3-4867-adc1-49a1e59f4673 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:41:45.063422 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-553bf3a0-ecc3-4867-adc1-49a1e59f4673 None admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:45.063915 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-553bf3a0-ecc3-4867-adc1-49a1e59f4673 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:45.064394 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-553bf3a0-ecc3-4867-adc1-49a1e59f4673 None admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:45.065244 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-553bf3a0-ecc3-4867-adc1-49a1e59f4673 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:41:45.066991 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-553bf3a0-ecc3-4867-adc1-49a1e59f4673 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:41:45.088164 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7086ea3-1327-44d9-b271-3c5049f5a4c2 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:41:45.107163 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:41:45.126628 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4286d6f9-4f22-4b88-8d1e-a5205e6a3a78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:41:45.127160 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4286d6f9-4f22-4b88-8d1e-a5205e6a3a78 None admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:45.127437 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4286d6f9-4f22-4b88-8d1e-a5205e6a3a78 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:45.127739 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4286d6f9-4f22-4b88-8d1e-a5205e6a3a78 None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:45.128415 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4286d6f9-4f22-4b88-8d1e-a5205e6a3a78 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:41:45.129751 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4286d6f9-4f22-4b88-8d1e-a5205e6a3a78 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:41:45.200740 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 16/34] 10.222.0.33 () {60 vars in 1131 bytes} [Sun Oct 9 00:41:44 2022] POST /identity/v3/roles => generated 219 bytes in 310 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:41:45.203739 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-553bf3a0-ecc3-4867-adc1-49a1e59f4673 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:41:45.249404 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 19/35] 10.222.0.33 () {60 vars in 1131 bytes} [Sun Oct 9 00:41:44 2022] POST /identity/v3/roles => generated 225 bytes in 290 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:41:45.250461 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4286d6f9-4f22-4b88-8d1e-a5205e6a3a78 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:41:45.353000 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-553bf3a0-ecc3-4867-adc1-49a1e59f4673 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:41:45.388476 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4286d6f9-4f22-4b88-8d1e-a5205e6a3a78 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:41:45.393112 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfbeb9d1-bb46-4d3c-a1c0-6674b15aeb95 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:41:45.393817 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfbeb9d1-bb46-4d3c-a1c0-6674b15aeb95 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:45.394097 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfbeb9d1-bb46-4d3c-a1c0-6674b15aeb95 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:45.394372 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfbeb9d1-bb46-4d3c-a1c0-6674b15aeb95 None admin] PATH_INFO: `/v3/users/d9c8218a78034540ba9ed09b697588a8` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:45.396082 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfbeb9d1-bb46-4d3c-a1c0-6674b15aeb95 None admin] RBAC: Authorizing `identity:get_user(user_id=d9c8218a78034540ba9ed09b697588a8)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:41:45.397647 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfbeb9d1-bb46-4d3c-a1c0-6674b15aeb95 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:41:45.399226 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 17/36] 10.222.0.33 () {56 vars in 1177 bytes} [Sun Oct 9 00:41:45 2022] GET /identity/v3/users/d9c8218a78034540ba9ed09b697588a8 => generated 247 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:41:45.400632 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b62fb4c-b18a-41bd-bd57-f99f787d15ba None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:45.400910 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b62fb4c-b18a-41bd-bd57-f99f787d15ba None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:45.401237 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b62fb4c-b18a-41bd-bd57-f99f787d15ba None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:45.402444 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 18/37] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:41:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:41:45.406127 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b62fb4c-b18a-41bd-bd57-f99f787d15ba None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:41:45.432644 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df93a131-7311-4d77-b251-d1b0a4f31704 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:41:45.433340 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df93a131-7311-4d77-b251-d1b0a4f31704 None admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:45.433779 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df93a131-7311-4d77-b251-d1b0a4f31704 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:45.434057 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df93a131-7311-4d77-b251-d1b0a4f31704 None admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:45.435304 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df93a131-7311-4d77-b251-d1b0a4f31704 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:41:45.436987 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df93a131-7311-4d77-b251-d1b0a4f31704 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:41:45.439836 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:41:45.486864 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 20/38] 10.222.0.33 () {60 vars in 1141 bytes} [Sun Oct 9 00:41:45 2022] POST /identity/v3/projects => generated 301 bytes in 237 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:41:45.488958 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38a285d8-ada8-4d9f-ba43-e7ca154c171e None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:45.489257 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38a285d8-ada8-4d9f-ba43-e7ca154c171e None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:45.489533 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38a285d8-ada8-4d9f-ba43-e7ca154c171e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:45.518251 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-38a285d8-ada8-4d9f-ba43-e7ca154c171e None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:41:45.521925 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-38a285d8-ada8-4d9f-ba43-e7ca154c171e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:41:45.560749 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:41:45.573856 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b62fb4c-b18a-41bd-bd57-f99f787d15ba None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:41:45.622702 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cbb3c4f-050e-42e1-bdd7-697d1b9aedb3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:41:45.623569 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cbb3c4f-050e-42e1-bdd7-697d1b9aedb3 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:45.623980 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cbb3c4f-050e-42e1-bdd7-697d1b9aedb3 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:45.624388 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cbb3c4f-050e-42e1-bdd7-697d1b9aedb3 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:45.628159 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cbb3c4f-050e-42e1-bdd7-697d1b9aedb3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:41:45.629655 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 21/39] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:41:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 09 00:41:45.630113 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cbb3c4f-050e-42e1-bdd7-697d1b9aedb3 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:41:45.632085 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 19/40] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:41:45 2022] GET /identity/v3/domains/default => generated 199 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:41:45.638359 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38a285d8-ada8-4d9f-ba43-e7ca154c171e None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:41:45.764106 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38a285d8-ada8-4d9f-ba43-e7ca154c171e None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:41:45.828722 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602a7423-6b0f-45ac-b215-fd7659f932dd None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:45.829053 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602a7423-6b0f-45ac-b215-fd7659f932dd None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:45.829283 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602a7423-6b0f-45ac-b215-fd7659f932dd None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:45.830671 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 20/41] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:41:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 09 00:41:45.836420 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19b6c543-17cf-484e-91ef-bc347de9bb2b None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:45.836739 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19b6c543-17cf-484e-91ef-bc347de9bb2b None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:45.836985 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19b6c543-17cf-484e-91ef-bc347de9bb2b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:45.857099 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-19b6c543-17cf-484e-91ef-bc347de9bb2b None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:41:45.858698 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-19b6c543-17cf-484e-91ef-bc347de9bb2b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:41:45.907925 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a18aa32-e985-4add-9b28-21ccd7989551 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:41:45.908583 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a18aa32-e985-4add-9b28-21ccd7989551 None admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:45.908866 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a18aa32-e985-4add-9b28-21ccd7989551 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:45.909144 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a18aa32-e985-4add-9b28-21ccd7989551 None admin] PATH_INFO: `/v3/domains/default/users/d9c8218a78034540ba9ed09b697588a8/roles/092036ee46ac427baf4d56468575f697` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:45.911031 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a18aa32-e985-4add-9b28-21ccd7989551 None admin] RBAC: Authorizing `identity:create_grant(domain_id=default, user_id=d9c8218a78034540ba9ed09b697588a8, role_id=092036ee46ac427baf4d56468575f697)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:41:45.912503 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a18aa32-e985-4add-9b28-21ccd7989551 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:41:45.929035 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 22/42] 10.222.0.33 () {58 vars in 1361 bytes} [Sun Oct 9 00:41:45 2022] PUT /identity/v3/domains/default/users/d9c8218a78034540ba9ed09b697588a8/roles/092036ee46ac427baf4d56468575f697 => generated 0 bytes in 292 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 09 00:41:45.931777 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a18aa32-e985-4add-9b28-21ccd7989551 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:41:45.936344 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 21/43] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:41:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 09 00:41:45.954232 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:41:46.047888 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a18aa32-e985-4add-9b28-21ccd7989551 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:41:46.092081 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab00e658-22a9-45de-b8fc-e0efc2cff4be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:41:46.092715 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab00e658-22a9-45de-b8fc-e0efc2cff4be None admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:46.094409 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab00e658-22a9-45de-b8fc-e0efc2cff4be None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:46.094409 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab00e658-22a9-45de-b8fc-e0efc2cff4be None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:46.094409 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab00e658-22a9-45de-b8fc-e0efc2cff4be None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:41:46.095305 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab00e658-22a9-45de-b8fc-e0efc2cff4be None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:41:46.112624 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 23/44] 10.222.0.33 () {60 vars in 1131 bytes} [Sun Oct 9 00:41:45 2022] POST /identity/v3/roles => generated 223 bytes in 181 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 09 00:41:46.210023 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19b6c543-17cf-484e-91ef-bc347de9bb2b None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:41:46.236773 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:41:46.321710 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19b6c543-17cf-484e-91ef-bc347de9bb2b None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:41:46.355798 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64df530f-ab41-40a5-a746-57c22ae73f2d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:41:46.356998 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64df530f-ab41-40a5-a746-57c22ae73f2d None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:46.357276 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64df530f-ab41-40a5-a746-57c22ae73f2d None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:46.357545 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64df530f-ab41-40a5-a746-57c22ae73f2d None admin] PATH_INFO: `/v3/domains/default` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:46.358635 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64df530f-ab41-40a5-a746-57c22ae73f2d None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:41:46.360105 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64df530f-ab41-40a5-a746-57c22ae73f2d None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:41:46.361871 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 22/45] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:41:46 2022] GET /identity/v3/domains/default => generated 199 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:41:46.367359 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab00e658-22a9-45de-b8fc-e0efc2cff4be None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:41:46.447562 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:41:46.470857 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab00e658-22a9-45de-b8fc-e0efc2cff4be None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:41:46.505451 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f66269b7-02ef-461a-a659-2c764a5f1659 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:41:46.506061 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66269b7-02ef-461a-a659-2c764a5f1659 None admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:46.506345 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66269b7-02ef-461a-a659-2c764a5f1659 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:46.506624 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66269b7-02ef-461a-a659-2c764a5f1659 None admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:46.507286 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f66269b7-02ef-461a-a659-2c764a5f1659 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:41:46.508790 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f66269b7-02ef-461a-a659-2c764a5f1659 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:41:46.535411 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 24/46] 10.222.0.33 () {60 vars in 1141 bytes} [Sun Oct 9 00:41:46 2022] POST /identity/v3/projects => generated 312 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:41:48.081394 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4e6f81-85b2-4256-bc4e-593a37dd5f64 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:48.081864 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4e6f81-85b2-4256-bc4e-593a37dd5f64 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:48.081982 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4e6f81-85b2-4256-bc4e-593a37dd5f64 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:48.083322 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 23/47] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:41:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:41:48.088706 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f01f339-44ef-496f-8df0-5d7ae286e4d3 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:48.089110 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f01f339-44ef-496f-8df0-5d7ae286e4d3 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:48.089387 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f01f339-44ef-496f-8df0-5d7ae286e4d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:48.108503 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-9f01f339-44ef-496f-8df0-5d7ae286e4d3 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:41:48.109690 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-9f01f339-44ef-496f-8df0-5d7ae286e4d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:41:48.177084 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 25/48] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:41:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 09 00:41:48.447774 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f4e6f81-85b2-4256-bc4e-593a37dd5f64 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:41:48.538156 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:41:48.549684 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f4e6f81-85b2-4256-bc4e-593a37dd5f64 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:41:48.583456 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f87268ce-4555-4492-a2d4-de99b982149b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:41:48.584138 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f87268ce-4555-4492-a2d4-de99b982149b None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:48.584419 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f87268ce-4555-4492-a2d4-de99b982149b None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:48.584691 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f87268ce-4555-4492-a2d4-de99b982149b None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:48.596370 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f87268ce-4555-4492-a2d4-de99b982149b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:41:48.601389 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-f87268ce-4555-4492-a2d4-de99b982149b None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 09 00:41:48.601389 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:41:48.601389 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:41:48.601389 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:41:48.601389 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:41:48.601389 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:41:48.601389 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:41:48.601389 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:41:48.601389 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:41:48.601389 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:41:48.601389 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:41:48.601389 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:41:48.601389 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:41:48.601389 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:41:48.601389 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:41:48.601389 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:41:48.602264 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:41:48.602264 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:41:48.602264 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:41:48.602264 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:41:48.602264 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:41:48.602264 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:41:48.602264 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:41:48.602264 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:41:48.602264 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:41:48.602264 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:41:48.602264 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:41:48.602264 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:41:48.602264 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:41:48.602264 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:41:48.602264 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:41:48.602264 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:41:48.603002 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:41:48.603002 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:41:48.603002 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:41:48.603002 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:41:48.603002 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:41:48.603002 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:41:48.603002 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:41:48.603002 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:41:48.603002 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:41:48.603002 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:41:48.603002 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 09 00:41:48.603002 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:41:48.603002 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 24/49] 10.222.0.33 () {56 vars in 1096 bytes} [Sun Oct 9 00:41:48 2022] GET /identity/v3/roles/admin => generated 83 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:41:48.609061 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f01f339-44ef-496f-8df0-5d7ae286e4d3 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:41:48.681817 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:41:48.696781 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ba4483-bddf-4ce5-bf3a-4c91ef5d065f None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:48.697069 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ba4483-bddf-4ce5-bf3a-4c91ef5d065f None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:48.697347 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ba4483-bddf-4ce5-bf3a-4c91ef5d065f None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:48.698731 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 25/50] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:41:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 09 00:41:48.703979 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0ed3ab9-cde2-4fca-bf22-ed964183510b None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:48.704265 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0ed3ab9-cde2-4fca-bf22-ed964183510b None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:48.704567 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0ed3ab9-cde2-4fca-bf22-ed964183510b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:48.722058 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-c0ed3ab9-cde2-4fca-bf22-ed964183510b None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:41:48.723245 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-c0ed3ab9-cde2-4fca-bf22-ed964183510b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:41:48.724388 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f01f339-44ef-496f-8df0-5d7ae286e4d3 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:41:48.761683 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-554fcef1-e9d6-4fe3-a772-31e7784e047a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:41:48.762356 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554fcef1-e9d6-4fe3-a772-31e7784e047a None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:48.762588 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554fcef1-e9d6-4fe3-a772-31e7784e047a None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:48.762864 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554fcef1-e9d6-4fe3-a772-31e7784e047a None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:48.763520 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-554fcef1-e9d6-4fe3-a772-31e7784e047a None admin] RBAC: Adding query filter params (name=admin) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:41:48.763855 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-554fcef1-e9d6-4fe3-a772-31e7784e047a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:41:48.765247 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-554fcef1-e9d6-4fe3-a772-31e7784e047a None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:41:48.773320 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 26/51] 10.222.0.33 () {56 vars in 1099 bytes} [Sun Oct 9 00:41:48 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:41:48.779436 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-554fcef1-e9d6-4fe3-a772-31e7784e047a None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:41:48.794469 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 26/52] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:41:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 09 00:41:48.874842 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-554fcef1-e9d6-4fe3-a772-31e7784e047a None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:41:48.895683 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:41:48.907711 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17273f8d-a813-4d26-ba6d-718003ab7532 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:41:48.908341 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17273f8d-a813-4d26-ba6d-718003ab7532 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:48.908627 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17273f8d-a813-4d26-ba6d-718003ab7532 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:48.908905 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17273f8d-a813-4d26-ba6d-718003ab7532 None admin] PATH_INFO: `/v3/users/d9c8218a78034540ba9ed09b697588a8` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:48.910029 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17273f8d-a813-4d26-ba6d-718003ab7532 None admin] RBAC: Authorizing `identity:get_user(user_id=d9c8218a78034540ba9ed09b697588a8)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:41:48.911483 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17273f8d-a813-4d26-ba6d-718003ab7532 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:41:48.913080 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 27/53] 10.222.0.33 () {56 vars in 1177 bytes} [Sun Oct 9 00:41:48 2022] GET /identity/v3/users/d9c8218a78034540ba9ed09b697588a8 => generated 247 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:41:48.919235 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0ed3ab9-cde2-4fca-bf22-ed964183510b None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:41:48.995725 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:41:49.013031 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0ed3ab9-cde2-4fca-bf22-ed964183510b None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:41:49.045609 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0689bab6-f96e-419f-98ca-654cf600afc4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:41:49.046225 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0689bab6-f96e-419f-98ca-654cf600afc4 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:49.046500 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0689bab6-f96e-419f-98ca-654cf600afc4 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:49.046768 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0689bab6-f96e-419f-98ca-654cf600afc4 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:49.047877 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0689bab6-f96e-419f-98ca-654cf600afc4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:41:49.049311 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0689bab6-f96e-419f-98ca-654cf600afc4 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:41:49.050911 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 27/54] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:41:48 2022] GET /identity/v3/domains/default => generated 199 bytes in 133 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:41:49.056528 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17273f8d-a813-4d26-ba6d-718003ab7532 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:41:49.066572 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0689bab6-f96e-419f-98ca-654cf600afc4 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:41:49.158188 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17273f8d-a813-4d26-ba6d-718003ab7532 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:41:49.165126 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0689bab6-f96e-419f-98ca-654cf600afc4 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:41:49.192775 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db0dade3-5323-44c7-ab63-68854ae383ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:41:49.193400 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0dade3-5323-44c7-ab63-68854ae383ea None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:49.193687 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0dade3-5323-44c7-ab63-68854ae383ea None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:49.193961 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0dade3-5323-44c7-ab63-68854ae383ea None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:49.194697 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db0dade3-5323-44c7-ab63-68854ae383ea None admin] RBAC: Adding query filter params (role.id=092036ee46ac427baf4d56468575f697, scope.domain.id=default, user.id=d9c8218a78034540ba9ed09b697588a8) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:41:49.195004 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db0dade3-5323-44c7-ab63-68854ae383ea None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:41:49.196528 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db0dade3-5323-44c7-ab63-68854ae383ea None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:41:49.203126 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 28/55] 10.222.0.33 () {56 vars in 1322 bytes} [Sun Oct 9 00:41:49 2022] GET /identity/v3/role_assignments?user.id=d9c8218a78034540ba9ed09b697588a8&scope.domain.id=default&role.id=092036ee46ac427baf4d56468575f697 => generated 530 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:41:49.306149 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71cdd651-982c-401e-bc47-510ebedf59d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:41:49.306707 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71cdd651-982c-401e-bc47-510ebedf59d1 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:49.306991 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71cdd651-982c-401e-bc47-510ebedf59d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:49.307267 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71cdd651-982c-401e-bc47-510ebedf59d1 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:49.308443 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71cdd651-982c-401e-bc47-510ebedf59d1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:41:49.310547 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71cdd651-982c-401e-bc47-510ebedf59d1 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:41:49.312193 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 28/56] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:41:49 2022] GET /identity/v3/domains/default => generated 199 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:41:49.318228 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db0dade3-5323-44c7-ab63-68854ae383ea None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:41:49.376450 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:41:49.422909 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db0dade3-5323-44c7-ab63-68854ae383ea None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:41:49.456556 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8775e98-222a-4423-983e-da065fe0a8c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:41:49.457162 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8775e98-222a-4423-983e-da065fe0a8c9 None admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:49.457446 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8775e98-222a-4423-983e-da065fe0a8c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:49.457723 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8775e98-222a-4423-983e-da065fe0a8c9 None admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:49.458376 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8775e98-222a-4423-983e-da065fe0a8c9 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:41:49.459852 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8775e98-222a-4423-983e-da065fe0a8c9 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:41:49.485302 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 29/57] 10.222.0.33 () {60 vars in 1140 bytes} [Sun Oct 9 00:41:49 2022] POST /identity/v3/projects => generated 298 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 09 00:41:51.601944 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7abfc1-95f5-48f0-8e99-057949e897a4 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:51.602459 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7abfc1-95f5-48f0-8e99-057949e897a4 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:51.602561 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7abfc1-95f5-48f0-8e99-057949e897a4 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:51.603852 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 29/58] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:41:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:41:51.608954 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b546919-1a4d-4f27-a6a1-3855dc156a37 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:51.609344 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b546919-1a4d-4f27-a6a1-3855dc156a37 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:51.609506 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b546919-1a4d-4f27-a6a1-3855dc156a37 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:51.627100 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-6b546919-1a4d-4f27-a6a1-3855dc156a37 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:41:51.628329 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-6b546919-1a4d-4f27-a6a1-3855dc156a37 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:41:51.692277 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 30/59] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:41:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 09 00:41:51.957059 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f7abfc1-95f5-48f0-8e99-057949e897a4 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:41:51.975829 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:41:52.055063 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f7abfc1-95f5-48f0-8e99-057949e897a4 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:41:52.088264 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e6d04e7-c139-47a0-8cf0-6fb6e8950555 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:41:52.088897 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e6d04e7-c139-47a0-8cf0-6fb6e8950555 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:52.089182 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e6d04e7-c139-47a0-8cf0-6fb6e8950555 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:52.089459 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e6d04e7-c139-47a0-8cf0-6fb6e8950555 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:52.090583 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e6d04e7-c139-47a0-8cf0-6fb6e8950555 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:41:52.092052 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e6d04e7-c139-47a0-8cf0-6fb6e8950555 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:41:52.093599 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 30/60] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:41:51 2022] GET /identity/v3/domains/default => generated 199 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:41:52.099045 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b546919-1a4d-4f27-a6a1-3855dc156a37 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:41:52.132352 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:41:52.196939 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b546919-1a4d-4f27-a6a1-3855dc156a37 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:41:52.228694 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b1e7498-9027-4c74-b87e-2b92a923a58a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:41:52.229232 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b1e7498-9027-4c74-b87e-2b92a923a58a None admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:52.229510 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b1e7498-9027-4c74-b87e-2b92a923a58a None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:52.229782 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b1e7498-9027-4c74-b87e-2b92a923a58a None admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:52.230429 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b1e7498-9027-4c74-b87e-2b92a923a58a None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:41:52.232390 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b1e7498-9027-4c74-b87e-2b92a923a58a None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:41:52.257682 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 31/61] 10.222.0.33 () {60 vars in 1132 bytes} [Sun Oct 9 00:41:52 2022] POST /identity/v3/users => generated 275 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:41:55.551445 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44b71a79-b6fc-4209-ac8c-11340c039596 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:55.552512 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44b71a79-b6fc-4209-ac8c-11340c039596 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:55.552993 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44b71a79-b6fc-4209-ac8c-11340c039596 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:55.554660 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 31/62] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:41:55 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:41:55.561166 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c9d6f86-daf1-46ae-a13a-6d7f7d88e92a None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:55.562109 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c9d6f86-daf1-46ae-a13a-6d7f7d88e92a None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:55.562577 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c9d6f86-daf1-46ae-a13a-6d7f7d88e92a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:55.592589 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-8c9d6f86-daf1-46ae-a13a-6d7f7d88e92a None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:41:55.594158 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-8c9d6f86-daf1-46ae-a13a-6d7f7d88e92a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:41:55.671794 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf70eb2b-af27-479d-afa4-68403200ea99 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:55.672410 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf70eb2b-af27-479d-afa4-68403200ea99 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:55.672895 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf70eb2b-af27-479d-afa4-68403200ea99 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:55.674411 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 32/63] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:41:55 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 09 00:41:55.679882 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbfa2b03-4146-4dd8-879f-0ff1d9403ec7 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:55.680408 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbfa2b03-4146-4dd8-879f-0ff1d9403ec7 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:55.680868 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbfa2b03-4146-4dd8-879f-0ff1d9403ec7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:55.706040 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-bbfa2b03-4146-4dd8-879f-0ff1d9403ec7 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:41:55.707577 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 32/64] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:41:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 09 00:41:55.709102 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-bbfa2b03-4146-4dd8-879f-0ff1d9403ec7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:41:55.804963 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 33/65] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:41:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 09 00:41:55.998594 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b038b54-6f10-48a7-9592-bbf566708adb None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:55.999264 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b038b54-6f10-48a7-9592-bbf566708adb None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:55.999763 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b038b54-6f10-48a7-9592-bbf566708adb None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:56.001292 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 33/66] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:41:55 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:41:56.007049 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14cb4b27-08e6-495a-9bb4-73c44bdff31a None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:56.007656 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14cb4b27-08e6-495a-9bb4-73c44bdff31a None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:56.008123 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14cb4b27-08e6-495a-9bb4-73c44bdff31a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:56.032722 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-14cb4b27-08e6-495a-9bb4-73c44bdff31a None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:41:56.033970 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-14cb4b27-08e6-495a-9bb4-73c44bdff31a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:41:56.047372 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:41:56.047692 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b038b54-6f10-48a7-9592-bbf566708adb None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:41:56.092162 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:41:56.126079 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 34/67] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:41:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 09 00:41:56.127685 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f12235a6-0831-4893-8191-6ced0ac8e805 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:56.128196 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f12235a6-0831-4893-8191-6ced0ac8e805 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:56.128684 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f12235a6-0831-4893-8191-6ced0ac8e805 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:56.130118 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 35/68] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:41:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:41:56.137876 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-739595c9-7532-4611-bbb3-9d5db9522763 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:56.138411 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-739595c9-7532-4611-bbb3-9d5db9522763 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:56.138902 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-739595c9-7532-4611-bbb3-9d5db9522763 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:56.159427 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-739595c9-7532-4611-bbb3-9d5db9522763 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:41:56.160657 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-739595c9-7532-4611-bbb3-9d5db9522763 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:41:56.201962 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b038b54-6f10-48a7-9592-bbf566708adb None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:41:56.232238 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 36/69] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:41:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 09 00:41:56.233562 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6a83f6b-8ce3-4942-a34a-ed0ed55145fa None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:56.233858 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6a83f6b-8ce3-4942-a34a-ed0ed55145fa None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:56.234135 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6a83f6b-8ce3-4942-a34a-ed0ed55145fa None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:56.235373 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 37/70] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:41:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:41:56.236282 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6a83f6b-8ce3-4942-a34a-ed0ed55145fa None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:41:56.242744 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-013f9fda-ee1c-42da-b39c-6bb3bdf319c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:41:56.243619 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-013f9fda-ee1c-42da-b39c-6bb3bdf319c8 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:56.243893 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-013f9fda-ee1c-42da-b39c-6bb3bdf319c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:56.244220 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-013f9fda-ee1c-42da-b39c-6bb3bdf319c8 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:56.254885 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-013f9fda-ee1c-42da-b39c-6bb3bdf319c8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:41:56.259719 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-013f9fda-ee1c-42da-b39c-6bb3bdf319c8 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 09 00:41:56.259719 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:41:56.259719 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:41:56.259719 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:41:56.259719 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:41:56.259719 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:41:56.259719 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:41:56.259719 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:41:56.259719 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:41:56.259719 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:41:56.259719 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:41:56.259719 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:41:56.259719 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:41:56.259719 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:41:56.259719 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:41:56.259719 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:41:56.260522 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:41:56.260522 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:41:56.260522 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:41:56.260522 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:41:56.260522 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:41:56.260522 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:41:56.260522 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:41:56.260522 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:41:56.260522 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:41:56.260522 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:41:56.260522 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:41:56.260522 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:41:56.260522 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:41:56.260522 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:41:56.260522 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:41:56.260522 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:41:56.261267 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:41:56.261267 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:41:56.261267 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:41:56.261267 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:41:56.261267 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:41:56.261267 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:41:56.261267 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:41:56.261267 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:41:56.261267 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:41:56.261267 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:41:56.261267 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 09 00:41:56.261267 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:41:56.261267 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 34/71] 10.222.0.33 () {56 vars in 1096 bytes} [Sun Oct 9 00:41:56 2022] GET /identity/v3/roles/admin => generated 83 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:41:56.263248 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33841463-7f8f-420b-8239-94117cd67475 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:56.263533 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33841463-7f8f-420b-8239-94117cd67475 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:56.263847 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33841463-7f8f-420b-8239-94117cd67475 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:56.281327 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-33841463-7f8f-420b-8239-94117cd67475 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:41:56.282529 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-33841463-7f8f-420b-8239-94117cd67475 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:41:56.331155 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:41:56.351799 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 35/72] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:41:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 09 00:41:56.360512 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33841463-7f8f-420b-8239-94117cd67475 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:41:56.361233 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6a83f6b-8ce3-4942-a34a-ed0ed55145fa None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:41:56.393443 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:41:56.399459 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb8aa90a-1fe5-45a7-877d-5cf55dc6b6f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:41:56.400268 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb8aa90a-1fe5-45a7-877d-5cf55dc6b6f0 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:56.400581 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb8aa90a-1fe5-45a7-877d-5cf55dc6b6f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:56.400860 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb8aa90a-1fe5-45a7-877d-5cf55dc6b6f0 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:56.412543 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb8aa90a-1fe5-45a7-877d-5cf55dc6b6f0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:41:56.418099 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-bb8aa90a-1fe5-45a7-877d-5cf55dc6b6f0 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 09 00:41:56.418099 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:41:56.418099 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:41:56.418099 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:41:56.418099 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:41:56.418099 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:41:56.418099 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:41:56.418099 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:41:56.418099 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:41:56.418099 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:41:56.418099 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:41:56.418099 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:41:56.418099 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:41:56.418099 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:41:56.418099 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:41:56.418099 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:41:56.418907 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:41:56.418907 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:41:56.418907 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:41:56.418907 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:41:56.418907 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:41:56.418907 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:41:56.418907 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:41:56.418907 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:41:56.418907 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:41:56.418907 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:41:56.418907 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:41:56.418907 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:41:56.418907 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:41:56.418907 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:41:56.418907 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:41:56.418907 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:41:56.420072 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:41:56.420072 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:41:56.420072 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:41:56.420072 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:41:56.420072 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:41:56.420072 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:41:56.420072 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:41:56.420072 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:41:56.420072 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:41:56.420072 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:41:56.420072 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 09 00:41:56.420072 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:41:56.420072 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 38/73] 10.222.0.33 () {56 vars in 1099 bytes} [Sun Oct 9 00:41:56 2022] GET /identity/v3/roles/member => generated 84 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:41:56.420673 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb8aa90a-1fe5-45a7-877d-5cf55dc6b6f0 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:41:56.517543 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33841463-7f8f-420b-8239-94117cd67475 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:41:56.536149 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb8aa90a-1fe5-45a7-877d-5cf55dc6b6f0 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:41:56.552501 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbde719c-d12b-444c-9a55-30488bb445ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:41:56.553408 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbde719c-d12b-444c-9a55-30488bb445ec None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:56.553515 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbde719c-d12b-444c-9a55-30488bb445ec None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:56.553755 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbde719c-d12b-444c-9a55-30488bb445ec None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:56.554418 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbde719c-d12b-444c-9a55-30488bb445ec None admin] RBAC: Adding query filter params (name=admin) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:41:56.554721 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbde719c-d12b-444c-9a55-30488bb445ec None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:41:56.556137 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbde719c-d12b-444c-9a55-30488bb445ec None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:41:56.565576 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 36/74] 10.222.0.33 () {56 vars in 1099 bytes} [Sun Oct 9 00:41:56 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:41:56.567011 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbde719c-d12b-444c-9a55-30488bb445ec None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:41:56.583287 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-178f0794-47fc-4d24-a7f1-4ab01a64fe0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:41:56.583878 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-178f0794-47fc-4d24-a7f1-4ab01a64fe0a None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:56.584185 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-178f0794-47fc-4d24-a7f1-4ab01a64fe0a None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:56.584437 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-178f0794-47fc-4d24-a7f1-4ab01a64fe0a None admin] PATH_INFO: `/v3/domains/default` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:56.585558 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-178f0794-47fc-4d24-a7f1-4ab01a64fe0a None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:41:56.587671 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-178f0794-47fc-4d24-a7f1-4ab01a64fe0a None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:41:56.591698 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 39/75] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:41:56 2022] GET /identity/v3/domains/default => generated 199 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:41:56.591698 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-178f0794-47fc-4d24-a7f1-4ab01a64fe0a None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:41:56.595885 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:41:56.645316 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:41:56.670971 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbde719c-d12b-444c-9a55-30488bb445ec None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:41:56.701102 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-178f0794-47fc-4d24-a7f1-4ab01a64fe0a None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:41:56.704919 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32ee6c56-3a53-49b1-84e9-698cd999a992 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:41:56.705517 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32ee6c56-3a53-49b1-84e9-698cd999a992 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:56.705795 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32ee6c56-3a53-49b1-84e9-698cd999a992 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:56.706066 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32ee6c56-3a53-49b1-84e9-698cd999a992 None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:56.706710 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32ee6c56-3a53-49b1-84e9-698cd999a992 None admin] RBAC: Adding query filter params (name=member) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:41:56.707007 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32ee6c56-3a53-49b1-84e9-698cd999a992 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:41:56.708417 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32ee6c56-3a53-49b1-84e9-698cd999a992 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:41:56.716195 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 37/76] 10.222.0.33 () {56 vars in 1101 bytes} [Sun Oct 9 00:41:56 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:41:56.716965 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32ee6c56-3a53-49b1-84e9-698cd999a992 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:41:56.738814 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fd93327-03e6-46a1-8b92-c8cd38af2a5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:41:56.739437 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fd93327-03e6-46a1-8b92-c8cd38af2a5e None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:56.739751 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fd93327-03e6-46a1-8b92-c8cd38af2a5e None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:56.740023 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fd93327-03e6-46a1-8b92-c8cd38af2a5e None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:56.750572 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fd93327-03e6-46a1-8b92-c8cd38af2a5e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:41:56.755344 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-0fd93327-03e6-46a1-8b92-c8cd38af2a5e None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 09 00:41:56.755344 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:41:56.755344 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:41:56.755344 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:41:56.755344 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:41:56.755344 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:41:56.755344 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:41:56.755344 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:41:56.755344 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:41:56.755344 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:41:56.755344 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:41:56.755344 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:41:56.755344 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:41:56.755344 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:41:56.755344 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:41:56.755344 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:41:56.756192 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:41:56.756192 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:41:56.756192 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:41:56.756192 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:41:56.756192 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:41:56.756192 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:41:56.756192 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:41:56.756192 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:41:56.756192 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:41:56.756192 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:41:56.756192 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:41:56.756192 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:41:56.756192 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:41:56.756192 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:41:56.756192 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:41:56.756192 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:41:56.756959 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:41:56.756959 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:41:56.756959 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:41:56.756959 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:41:56.756959 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:41:56.756959 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:41:56.756959 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:41:56.756959 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:41:56.756959 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:41:56.756959 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:41:56.756959 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 09 00:41:56.756959 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:41:56.756959 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 40/77] 10.222.0.33 () {56 vars in 1114 bytes} [Sun Oct 9 00:41:56 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:41:56.757574 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fd93327-03e6-46a1-8b92-c8cd38af2a5e None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:41:56.818102 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:41:56.819655 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32ee6c56-3a53-49b1-84e9-698cd999a992 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:41:56.852246 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-540280b7-65d6-4601-9865-fd6237fead47 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:41:56.852803 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-540280b7-65d6-4601-9865-fd6237fead47 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:56.853086 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-540280b7-65d6-4601-9865-fd6237fead47 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:56.853363 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-540280b7-65d6-4601-9865-fd6237fead47 None admin] PATH_INFO: `/v3/users/d9c8218a78034540ba9ed09b697588a8` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:56.854486 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-540280b7-65d6-4601-9865-fd6237fead47 None admin] RBAC: Authorizing `identity:get_user(user_id=d9c8218a78034540ba9ed09b697588a8)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:41:56.855799 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fd93327-03e6-46a1-8b92-c8cd38af2a5e None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:41:56.856495 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-540280b7-65d6-4601-9865-fd6237fead47 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:41:56.858479 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 38/78] 10.222.0.33 () {56 vars in 1177 bytes} [Sun Oct 9 00:41:56 2022] GET /identity/v3/users/d9c8218a78034540ba9ed09b697588a8 => generated 247 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:41:56.858837 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-540280b7-65d6-4601-9865-fd6237fead47 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:41:56.863789 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:41:56.889438 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88449a62-1f74-4761-b01b-bb17bb0aa3a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:41:56.890044 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88449a62-1f74-4761-b01b-bb17bb0aa3a5 None admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:56.890326 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88449a62-1f74-4761-b01b-bb17bb0aa3a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:56.890600 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88449a62-1f74-4761-b01b-bb17bb0aa3a5 None admin] PATH_INFO: `/v3/users` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:56.891327 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88449a62-1f74-4761-b01b-bb17bb0aa3a5 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:41:56.892793 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88449a62-1f74-4761-b01b-bb17bb0aa3a5 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:41:56.921838 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 41/79] 10.222.0.33 () {60 vars in 1132 bytes} [Sun Oct 9 00:41:56 2022] POST /identity/v3/users => generated 289 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:41:56.922655 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88449a62-1f74-4761-b01b-bb17bb0aa3a5 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:41:56.979513 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-540280b7-65d6-4601-9865-fd6237fead47 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:41:57.018618 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89a7cb64-bdf6-4b71-a0d3-7ba85f45ee24 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:41:57.019268 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a7cb64-bdf6-4b71-a0d3-7ba85f45ee24 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:57.019555 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a7cb64-bdf6-4b71-a0d3-7ba85f45ee24 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:57.019866 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a7cb64-bdf6-4b71-a0d3-7ba85f45ee24 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:57.033205 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89a7cb64-bdf6-4b71-a0d3-7ba85f45ee24 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:41:57.037847 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88449a62-1f74-4761-b01b-bb17bb0aa3a5 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:41:57.038645 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-89a7cb64-bdf6-4b71-a0d3-7ba85f45ee24 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 09 00:41:57.038645 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:41:57.038645 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:41:57.038645 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:41:57.038645 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:41:57.038645 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:41:57.038645 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:41:57.038645 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:41:57.038645 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:41:57.038645 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:41:57.038645 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:41:57.038645 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:41:57.038645 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:41:57.038645 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:41:57.038645 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:41:57.038645 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:41:57.039473 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:41:57.039473 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:41:57.039473 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:41:57.039473 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:41:57.039473 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:41:57.039473 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:41:57.039473 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:41:57.039473 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:41:57.039473 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:41:57.039473 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:41:57.039473 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:41:57.039473 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:41:57.039473 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:41:57.039473 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:41:57.039473 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:41:57.039473 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:41:57.040533 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:41:57.040533 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:41:57.040533 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:41:57.040533 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:41:57.040533 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:41:57.040533 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:41:57.040533 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:41:57.040533 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:41:57.040533 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:41:57.040533 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:41:57.040533 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 09 00:41:57.040533 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:41:57.040533 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 39/80] 10.222.0.33 () {56 vars in 1099 bytes} [Sun Oct 9 00:41:56 2022] GET /identity/v3/roles/member => generated 84 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:41:57.041170 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89a7cb64-bdf6-4b71-a0d3-7ba85f45ee24 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:41:57.065294 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:41:57.090561 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8229bf9f-ef75-4f4c-9654-b0949104480b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:41:57.091869 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8229bf9f-ef75-4f4c-9654-b0949104480b None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:57.092434 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8229bf9f-ef75-4f4c-9654-b0949104480b None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:57.092870 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8229bf9f-ef75-4f4c-9654-b0949104480b None admin] PATH_INFO: `/v3/users/da5fb3b5405648df8a2114ff2dca2d02` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:57.113298 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8229bf9f-ef75-4f4c-9654-b0949104480b None admin] RBAC: Authorizing `identity:get_user(user_id=da5fb3b5405648df8a2114ff2dca2d02)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:41:57.114765 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8229bf9f-ef75-4f4c-9654-b0949104480b None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:41:57.115187 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:41:57.116545 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 42/81] 10.222.0.33 () {56 vars in 1177 bytes} [Sun Oct 9 00:41:56 2022] GET /identity/v3/users/da5fb3b5405648df8a2114ff2dca2d02 => generated 275 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:41:57.117541 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8229bf9f-ef75-4f4c-9654-b0949104480b None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:41:57.163969 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89a7cb64-bdf6-4b71-a0d3-7ba85f45ee24 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:41:57.209238 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc778ae-1f31-4f66-ba1b-63190c4b3bcb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:41:57.209856 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc778ae-1f31-4f66-ba1b-63190c4b3bcb None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:57.210136 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc778ae-1f31-4f66-ba1b-63190c4b3bcb None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:57.210407 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc778ae-1f31-4f66-ba1b-63190c4b3bcb None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:57.211075 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbc778ae-1f31-4f66-ba1b-63190c4b3bcb None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:41:57.211376 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbc778ae-1f31-4f66-ba1b-63190c4b3bcb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:41:57.212808 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbc778ae-1f31-4f66-ba1b-63190c4b3bcb None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:41:57.224408 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 40/82] 10.222.0.33 () {56 vars in 1111 bytes} [Sun Oct 9 00:41:57 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:41:57.225378 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc778ae-1f31-4f66-ba1b-63190c4b3bcb None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:41:57.237458 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8229bf9f-ef75-4f4c-9654-b0949104480b None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:41:57.277236 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b87ce8e1-a734-44ce-b0f6-1263ae3c4bad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:41:57.277866 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b87ce8e1-a734-44ce-b0f6-1263ae3c4bad None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:57.278201 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b87ce8e1-a734-44ce-b0f6-1263ae3c4bad None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:57.278481 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b87ce8e1-a734-44ce-b0f6-1263ae3c4bad None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:57.336069 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:41:57.348682 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc778ae-1f31-4f66-ba1b-63190c4b3bcb None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:41:57.357247 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b87ce8e1-a734-44ce-b0f6-1263ae3c4bad None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:41:57.358749 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b87ce8e1-a734-44ce-b0f6-1263ae3c4bad None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:41:57.377118 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:41:57.388809 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81de7a37-2a85-4903-b6fa-62f8968e6f68 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:41:57.389990 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81de7a37-2a85-4903-b6fa-62f8968e6f68 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:57.390506 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81de7a37-2a85-4903-b6fa-62f8968e6f68 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:57.391024 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81de7a37-2a85-4903-b6fa-62f8968e6f68 None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:57.391988 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81de7a37-2a85-4903-b6fa-62f8968e6f68 None admin] RBAC: Adding query filter params (name=member) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:41:57.392488 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81de7a37-2a85-4903-b6fa-62f8968e6f68 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:41:57.394122 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81de7a37-2a85-4903-b6fa-62f8968e6f68 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:41:57.403231 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 41/83] 10.222.0.33 () {56 vars in 1101 bytes} [Sun Oct 9 00:41:57 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:41:57.404795 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81de7a37-2a85-4903-b6fa-62f8968e6f68 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:41:57.470385 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 43/84] 10.222.0.33 () {58 vars in 1282 bytes} [Sun Oct 9 00:41:57 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 353 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 09 00:41:57.475380 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b87ce8e1-a734-44ce-b0f6-1263ae3c4bad None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:41:57.514719 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81de7a37-2a85-4903-b6fa-62f8968e6f68 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:41:57.562048 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04639592-03dd-412b-94e3-23a0f7d6b60b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:41:57.562660 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04639592-03dd-412b-94e3-23a0f7d6b60b None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:57.562944 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04639592-03dd-412b-94e3-23a0f7d6b60b None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:57.563219 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04639592-03dd-412b-94e3-23a0f7d6b60b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:57.566415 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:41:57.596266 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b87ce8e1-a734-44ce-b0f6-1263ae3c4bad None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:41:57.629770 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:41:57.634709 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd571675-fc46-429f-8899-491a813a2561 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:41:57.635426 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd571675-fc46-429f-8899-491a813a2561 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:57.635817 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd571675-fc46-429f-8899-491a813a2561 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:57.636153 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd571675-fc46-429f-8899-491a813a2561 None admin] PATH_INFO: `/v3/users/da5fb3b5405648df8a2114ff2dca2d02` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:57.638510 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd571675-fc46-429f-8899-491a813a2561 None admin] RBAC: Authorizing `identity:get_user(user_id=da5fb3b5405648df8a2114ff2dca2d02)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:41:57.640675 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd571675-fc46-429f-8899-491a813a2561 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:41:57.642685 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 44/85] 10.222.0.33 () {56 vars in 1177 bytes} [Sun Oct 9 00:41:57 2022] GET /identity/v3/users/da5fb3b5405648df8a2114ff2dca2d02 => generated 275 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:41:57.643362 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd571675-fc46-429f-8899-491a813a2561 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:41:57.643823 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04639592-03dd-412b-94e3-23a0f7d6b60b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:41:57.645382 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04639592-03dd-412b-94e3-23a0f7d6b60b None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:41:57.753604 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 42/86] 10.222.0.33 () {58 vars in 1282 bytes} [Sun Oct 9 00:41:57 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 349 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 09 00:41:57.754565 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04639592-03dd-412b-94e3-23a0f7d6b60b None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:41:57.754927 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd571675-fc46-429f-8899-491a813a2561 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:41:57.793465 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ee67385-a0c7-4e1c-b15b-66b7dcbbdc00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:41:57.794252 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee67385-a0c7-4e1c-b15b-66b7dcbbdc00 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:57.794639 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee67385-a0c7-4e1c-b15b-66b7dcbbdc00 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:57.795014 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee67385-a0c7-4e1c-b15b-66b7dcbbdc00 None admin] PATH_INFO: `/v3/users/da5fb3b5405648df8a2114ff2dca2d02` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:57.796276 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ee67385-a0c7-4e1c-b15b-66b7dcbbdc00 None admin] RBAC: Authorizing `identity:get_user(user_id=da5fb3b5405648df8a2114ff2dca2d02)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:41:57.797941 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ee67385-a0c7-4e1c-b15b-66b7dcbbdc00 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:41:57.800253 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 45/87] 10.222.0.33 () {56 vars in 1177 bytes} [Sun Oct 9 00:41:57 2022] GET /identity/v3/users/da5fb3b5405648df8a2114ff2dca2d02 => generated 275 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:41:57.801306 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ee67385-a0c7-4e1c-b15b-66b7dcbbdc00 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:41:57.817658 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:41:57.851070 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:41:57.866556 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04639592-03dd-412b-94e3-23a0f7d6b60b None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:41:57.904178 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e5f4620-0ef7-4965-bcc9-239d6a92528d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:41:57.904667 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e5f4620-0ef7-4965-bcc9-239d6a92528d None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:57.904952 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e5f4620-0ef7-4965-bcc9-239d6a92528d None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:57.905231 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e5f4620-0ef7-4965-bcc9-239d6a92528d None admin] PATH_INFO: `/v3/projects/768d646302f443f1a7d490b8c3d4bdbb` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:57.906361 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e5f4620-0ef7-4965-bcc9-239d6a92528d None admin] RBAC: Authorizing `identity:get_project(project_id=768d646302f443f1a7d490b8c3d4bdbb)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:41:57.907864 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e5f4620-0ef7-4965-bcc9-239d6a92528d None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:41:57.909987 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 43/88] 10.222.0.33 () {56 vars in 1186 bytes} [Sun Oct 9 00:41:57 2022] GET /identity/v3/projects/768d646302f443f1a7d490b8c3d4bdbb => generated 298 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:41:57.910778 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e5f4620-0ef7-4965-bcc9-239d6a92528d None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:41:57.924893 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ee67385-a0c7-4e1c-b15b-66b7dcbbdc00 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:41:57.969848 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80d0d7d2-0862-4cb0-a071-c6812b30d810 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:41:57.971353 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80d0d7d2-0862-4cb0-a071-c6812b30d810 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:57.971944 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80d0d7d2-0862-4cb0-a071-c6812b30d810 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:57.972699 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80d0d7d2-0862-4cb0-a071-c6812b30d810 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:58.021331 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e5f4620-0ef7-4965-bcc9-239d6a92528d None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:41:58.052041 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80d0d7d2-0862-4cb0-a071-c6812b30d810 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:41:58.054627 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80d0d7d2-0862-4cb0-a071-c6812b30d810 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:41:58.059308 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eb0e726-cce2-4a25-b2ca-2da8e6bbbef9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:41:58.059969 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb0e726-cce2-4a25-b2ca-2da8e6bbbef9 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:58.060251 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb0e726-cce2-4a25-b2ca-2da8e6bbbef9 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:58.060525 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb0e726-cce2-4a25-b2ca-2da8e6bbbef9 None admin] PATH_INFO: `/v3/projects/768d646302f443f1a7d490b8c3d4bdbb` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:58.061660 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eb0e726-cce2-4a25-b2ca-2da8e6bbbef9 None admin] RBAC: Authorizing `identity:get_project(project_id=768d646302f443f1a7d490b8c3d4bdbb)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:41:58.063169 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eb0e726-cce2-4a25-b2ca-2da8e6bbbef9 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:41:58.065448 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 44/89] 10.222.0.33 () {56 vars in 1186 bytes} [Sun Oct 9 00:41:57 2022] GET /identity/v3/projects/768d646302f443f1a7d490b8c3d4bdbb => generated 298 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:41:58.065850 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eb0e726-cce2-4a25-b2ca-2da8e6bbbef9 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:41:58.074275 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:41:58.098295 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:41:58.176734 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 46/90] 10.222.0.33 () {58 vars in 1282 bytes} [Sun Oct 9 00:41:57 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 376 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 09 00:41:58.177403 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80d0d7d2-0862-4cb0-a071-c6812b30d810 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:41:58.192575 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eb0e726-cce2-4a25-b2ca-2da8e6bbbef9 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:41:58.239689 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb2e66c9-d80b-4dee-b13b-3c87a679010e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:41:58.240950 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2e66c9-d80b-4dee-b13b-3c87a679010e None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:58.241682 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2e66c9-d80b-4dee-b13b-3c87a679010e None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:58.242569 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2e66c9-d80b-4dee-b13b-3c87a679010e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:58.297595 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80d0d7d2-0862-4cb0-a071-c6812b30d810 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:41:58.314971 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:41:58.317511 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb2e66c9-d80b-4dee-b13b-3c87a679010e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:41:58.319037 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb2e66c9-d80b-4dee-b13b-3c87a679010e None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:41:58.342223 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03287414-5c9e-4744-a5c4-f0f182f16ea1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:41:58.343159 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03287414-5c9e-4744-a5c4-f0f182f16ea1 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:58.343627 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03287414-5c9e-4744-a5c4-f0f182f16ea1 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:58.344017 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:41:58.345151 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03287414-5c9e-4744-a5c4-f0f182f16ea1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:58.346021 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03287414-5c9e-4744-a5c4-f0f182f16ea1 None admin] RBAC: Adding query filter params (role.id=092036ee46ac427baf4d56468575f697, scope.project.id=768d646302f443f1a7d490b8c3d4bdbb, user.id=d9c8218a78034540ba9ed09b697588a8) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:41:58.346621 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03287414-5c9e-4744-a5c4-f0f182f16ea1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:41:58.349106 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03287414-5c9e-4744-a5c4-f0f182f16ea1 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:41:58.361429 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 47/91] 10.222.0.33 () {56 vars in 1374 bytes} [Sun Oct 9 00:41:58 2022] GET /identity/v3/role_assignments?user.id=d9c8218a78034540ba9ed09b697588a8&scope.project.id=768d646302f443f1a7d490b8c3d4bdbb&role.id=092036ee46ac427baf4d56468575f697 => generated 259 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:41:58.363023 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03287414-5c9e-4744-a5c4-f0f182f16ea1 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:41:58.428602 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 45/92] 10.222.0.33 () {58 vars in 1282 bytes} [Sun Oct 9 00:41:58 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 363 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 09 00:41:58.429136 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb2e66c9-d80b-4dee-b13b-3c87a679010e None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:41:58.489321 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03287414-5c9e-4744-a5c4-f0f182f16ea1 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:41:58.537218 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6382f0c0-b8bd-4a66-bb75-b6ff33946349 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:41:58.537660 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6382f0c0-b8bd-4a66-bb75-b6ff33946349 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:58.539625 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6382f0c0-b8bd-4a66-bb75-b6ff33946349 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:58.539625 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6382f0c0-b8bd-4a66-bb75-b6ff33946349 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:58.539625 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6382f0c0-b8bd-4a66-bb75-b6ff33946349 None admin] RBAC: Adding query filter params (role.id=588016933bab436d8b7c3ac76827356d, scope.project.id=768d646302f443f1a7d490b8c3d4bdbb, user.id=da5fb3b5405648df8a2114ff2dca2d02) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:41:58.539625 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6382f0c0-b8bd-4a66-bb75-b6ff33946349 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:41:58.541030 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6382f0c0-b8bd-4a66-bb75-b6ff33946349 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:41:58.558863 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 48/93] 10.222.0.33 () {56 vars in 1374 bytes} [Sun Oct 9 00:41:58 2022] GET /identity/v3/role_assignments?user.id=da5fb3b5405648df8a2114ff2dca2d02&scope.project.id=768d646302f443f1a7d490b8c3d4bdbb&role.id=588016933bab436d8b7c3ac76827356d => generated 259 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:41:58.559766 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6382f0c0-b8bd-4a66-bb75-b6ff33946349 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:41:58.569130 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb2e66c9-d80b-4dee-b13b-3c87a679010e None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:41:58.626069 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:41:58.648761 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2333e05-b4a3-4234-8870-d03a41b9d9bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:41:58.650767 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2333e05-b4a3-4234-8870-d03a41b9d9bc None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:58.651339 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2333e05-b4a3-4234-8870-d03a41b9d9bc None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:58.653661 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2333e05-b4a3-4234-8870-d03a41b9d9bc None admin] PATH_INFO: `/v3/projects/768d646302f443f1a7d490b8c3d4bdbb` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:58.655550 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:41:58.657380 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2333e05-b4a3-4234-8870-d03a41b9d9bc None admin] RBAC: Authorizing `identity:get_project(project_id=768d646302f443f1a7d490b8c3d4bdbb)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:41:58.659410 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2333e05-b4a3-4234-8870-d03a41b9d9bc None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:41:58.662760 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 46/94] 10.222.0.33 () {56 vars in 1186 bytes} [Sun Oct 9 00:41:58 2022] GET /identity/v3/projects/768d646302f443f1a7d490b8c3d4bdbb => generated 298 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:41:58.669623 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2333e05-b4a3-4234-8870-d03a41b9d9bc None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:41:58.811958 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6382f0c0-b8bd-4a66-bb75-b6ff33946349 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:41:58.862737 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab064e2c-75da-4883-ab0b-b3672b800b58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:41:58.863383 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab064e2c-75da-4883-ab0b-b3672b800b58 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:58.863704 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab064e2c-75da-4883-ab0b-b3672b800b58 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:58.863984 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab064e2c-75da-4883-ab0b-b3672b800b58 None admin] PATH_INFO: `/v3/projects/fc09cf0f89194e89b47f7e4e63ad25da` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:58.865109 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab064e2c-75da-4883-ab0b-b3672b800b58 None admin] RBAC: Authorizing `identity:get_project(project_id=fc09cf0f89194e89b47f7e4e63ad25da)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:41:58.866570 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab064e2c-75da-4883-ab0b-b3672b800b58 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:41:58.868423 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 49/95] 10.222.0.33 () {56 vars in 1186 bytes} [Sun Oct 9 00:41:58 2022] GET /identity/v3/projects/fc09cf0f89194e89b47f7e4e63ad25da => generated 312 bytes in 309 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:41:58.874579 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab064e2c-75da-4883-ab0b-b3672b800b58 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:41:58.883332 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2333e05-b4a3-4234-8870-d03a41b9d9bc None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:41:58.968332 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-731f0e68-5748-43b0-b86d-44396576879e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:41:58.969276 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-731f0e68-5748-43b0-b86d-44396576879e None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:58.969736 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-731f0e68-5748-43b0-b86d-44396576879e None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:58.970183 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-731f0e68-5748-43b0-b86d-44396576879e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:58.971079 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-731f0e68-5748-43b0-b86d-44396576879e None admin] RBAC: Adding query filter params (role.id=7c1fdac272a34be99661077b0f1b8014, scope.project.id=768d646302f443f1a7d490b8c3d4bdbb, user.id=da5fb3b5405648df8a2114ff2dca2d02) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:41:58.971585 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-731f0e68-5748-43b0-b86d-44396576879e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:41:58.973241 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-731f0e68-5748-43b0-b86d-44396576879e None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:41:58.985470 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 47/96] 10.222.0.33 () {56 vars in 1374 bytes} [Sun Oct 9 00:41:58 2022] GET /identity/v3/role_assignments?user.id=da5fb3b5405648df8a2114ff2dca2d02&scope.project.id=768d646302f443f1a7d490b8c3d4bdbb&role.id=7c1fdac272a34be99661077b0f1b8014 => generated 259 bytes in 317 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:41:59.054814 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab064e2c-75da-4883-ab0b-b3672b800b58 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:41:59.083992 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:41:59.121834 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22a42d56-5fb5-46e6-9572-17f9e60ed268 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:41:59.122459 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a42d56-5fb5-46e6-9572-17f9e60ed268 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:59.122743 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a42d56-5fb5-46e6-9572-17f9e60ed268 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:59.123015 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a42d56-5fb5-46e6-9572-17f9e60ed268 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:59.123780 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22a42d56-5fb5-46e6-9572-17f9e60ed268 None admin] RBAC: Adding query filter params (role.id=588016933bab436d8b7c3ac76827356d, scope.project.id=fc09cf0f89194e89b47f7e4e63ad25da, user.id=da5fb3b5405648df8a2114ff2dca2d02) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:41:59.124093 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22a42d56-5fb5-46e6-9572-17f9e60ed268 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:41:59.125569 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22a42d56-5fb5-46e6-9572-17f9e60ed268 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:41:59.137217 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 50/97] 10.222.0.33 () {56 vars in 1374 bytes} [Sun Oct 9 00:41:58 2022] GET /identity/v3/role_assignments?user.id=da5fb3b5405648df8a2114ff2dca2d02&scope.project.id=fc09cf0f89194e89b47f7e4e63ad25da&role.id=588016933bab436d8b7c3ac76827356d => generated 259 bytes in 263 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:41:59.540548 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c14f88-e20b-4b52-9d5f-2c0c0b4e0646 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:59.541312 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c14f88-e20b-4b52-9d5f-2c0c0b4e0646 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:59.541861 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c14f88-e20b-4b52-9d5f-2c0c0b4e0646 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:59.543514 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 48/98] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:41:59 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:41:59.548822 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f7af72-3300-41be-a6e6-9184fdd0f53e None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:59.549353 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f7af72-3300-41be-a6e6-9184fdd0f53e None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:59.549826 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f7af72-3300-41be-a6e6-9184fdd0f53e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:59.576583 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-c9f7af72-3300-41be-a6e6-9184fdd0f53e None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:41:59.578098 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-c9f7af72-3300-41be-a6e6-9184fdd0f53e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:41:59.649683 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 51/99] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:41:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 09 00:41:59.659242 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32123669-73e4-4813-bd93-afede7547f43 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:59.660189 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32123669-73e4-4813-bd93-afede7547f43 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:59.660684 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32123669-73e4-4813-bd93-afede7547f43 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:59.662303 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 49/100] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:41:59 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 09 00:41:59.668430 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8bd2e97-95f1-458b-90c8-e5d0da593499 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:41:59.669247 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8bd2e97-95f1-458b-90c8-e5d0da593499 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:41:59.669717 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8bd2e97-95f1-458b-90c8-e5d0da593499 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:41:59.693846 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-d8bd2e97-95f1-458b-90c8-e5d0da593499 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:41:59.695355 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-d8bd2e97-95f1-458b-90c8-e5d0da593499 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:41:59.779822 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 52/101] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:41:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 09 00:42:00.118732 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32123669-73e4-4813-bd93-afede7547f43 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:00.191479 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:00.248766 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32123669-73e4-4813-bd93-afede7547f43 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:00.287610 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2dcd12f-f7fe-47ba-a14b-105cd44f7962 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:00.288258 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2dcd12f-f7fe-47ba-a14b-105cd44f7962 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:00.288541 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2dcd12f-f7fe-47ba-a14b-105cd44f7962 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:00.288818 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2dcd12f-f7fe-47ba-a14b-105cd44f7962 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:00.299883 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2dcd12f-f7fe-47ba-a14b-105cd44f7962 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:42:00.304753 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-e2dcd12f-f7fe-47ba-a14b-105cd44f7962 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 09 00:42:00.304753 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:42:00.304753 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:42:00.304753 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:42:00.304753 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:42:00.304753 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:42:00.304753 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:42:00.304753 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:42:00.304753 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:42:00.304753 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:42:00.304753 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:42:00.304753 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:42:00.304753 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:42:00.304753 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:42:00.304753 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:42:00.304753 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:42:00.305588 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:42:00.305588 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:42:00.305588 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:42:00.305588 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:42:00.305588 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:42:00.305588 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:42:00.305588 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:42:00.305588 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:42:00.305588 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:42:00.305588 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:42:00.305588 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:42:00.305588 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:42:00.305588 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:42:00.305588 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:42:00.305588 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:42:00.305588 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:42:00.306700 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:42:00.306700 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:42:00.306700 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:42:00.306700 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:42:00.306700 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:42:00.306700 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:42:00.306700 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:42:00.306700 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:42:00.306700 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:42:00.306700 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:42:00.306700 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 09 00:42:00.306700 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:42:00.306700 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 50/102] 10.222.0.33 () {56 vars in 1099 bytes} [Sun Oct 9 00:42:00 2022] GET /identity/v3/roles/reader => generated 84 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:42:00.312337 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8bd2e97-95f1-458b-90c8-e5d0da593499 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:00.347484 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2dcd12f-f7fe-47ba-a14b-105cd44f7962 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:00.350822 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:00.428602 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8bd2e97-95f1-458b-90c8-e5d0da593499 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:00.472737 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea511a6c-5789-44d9-8654-63c6c2b3cfd7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:00.473472 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea511a6c-5789-44d9-8654-63c6c2b3cfd7 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:00.473762 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea511a6c-5789-44d9-8654-63c6c2b3cfd7 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:00.474039 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea511a6c-5789-44d9-8654-63c6c2b3cfd7 None admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:00.474705 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea511a6c-5789-44d9-8654-63c6c2b3cfd7 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:00.475011 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea511a6c-5789-44d9-8654-63c6c2b3cfd7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:00.476449 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea511a6c-5789-44d9-8654-63c6c2b3cfd7 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:00.485270 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 53/103] 10.222.0.33 () {56 vars in 1101 bytes} [Sun Oct 9 00:42:00 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:00.490991 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea511a6c-5789-44d9-8654-63c6c2b3cfd7 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:00.546371 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2dcd12f-f7fe-47ba-a14b-105cd44f7962 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:00.571634 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:00.592195 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29cca6c3-27b5-4cd2-bce3-cfb2e7f76f65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:00.592830 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29cca6c3-27b5-4cd2-bce3-cfb2e7f76f65 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:00.593112 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29cca6c3-27b5-4cd2-bce3-cfb2e7f76f65 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:00.593386 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29cca6c3-27b5-4cd2-bce3-cfb2e7f76f65 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:00.594586 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29cca6c3-27b5-4cd2-bce3-cfb2e7f76f65 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:00.596123 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29cca6c3-27b5-4cd2-bce3-cfb2e7f76f65 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:00.597741 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 51/104] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:42:00 2022] GET /identity/v3/domains/default => generated 199 bytes in 251 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:00.603031 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29cca6c3-27b5-4cd2-bce3-cfb2e7f76f65 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:00.693528 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:00.701893 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea511a6c-5789-44d9-8654-63c6c2b3cfd7 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:00.709027 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29cca6c3-27b5-4cd2-bce3-cfb2e7f76f65 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:00.746686 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b206590b-094a-4aa3-9f91-dfece082456e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:00.747410 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b206590b-094a-4aa3-9f91-dfece082456e None admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:00.747734 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b206590b-094a-4aa3-9f91-dfece082456e None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:00.748129 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b206590b-094a-4aa3-9f91-dfece082456e None admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:00.748810 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b206590b-094a-4aa3-9f91-dfece082456e None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:00.750308 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b206590b-094a-4aa3-9f91-dfece082456e None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:00.784901 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61d8310d-1a41-42e3-80d5-8e28f35f6b67 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:00.787444 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 52/105] 10.222.0.33 () {60 vars in 1141 bytes} [Sun Oct 9 00:42:00 2022] POST /identity/v3/projects => generated 302 bytes in 185 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 09 00:42:00.787726 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d8310d-1a41-42e3-80d5-8e28f35f6b67 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:00.787976 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d8310d-1a41-42e3-80d5-8e28f35f6b67 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:00.788263 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d8310d-1a41-42e3-80d5-8e28f35f6b67 None admin] PATH_INFO: `/v3/users/d90f753f3b2e4bb08eb926d2d604138e` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:00.804545 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61d8310d-1a41-42e3-80d5-8e28f35f6b67 None admin] RBAC: Authorizing `identity:get_user(user_id=d90f753f3b2e4bb08eb926d2d604138e)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:00.806161 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61d8310d-1a41-42e3-80d5-8e28f35f6b67 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:00.808484 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 54/106] 10.222.0.33 () {56 vars in 1177 bytes} [Sun Oct 9 00:42:00 2022] GET /identity/v3/users/d90f753f3b2e4bb08eb926d2d604138e => generated 289 bytes in 318 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:42:00.814150 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b206590b-094a-4aa3-9f91-dfece082456e None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:00.857085 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:00.934438 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b206590b-094a-4aa3-9f91-dfece082456e None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:00.971942 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b299cae4-892d-47bf-bedd-945aba4e9f12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:00.972573 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b299cae4-892d-47bf-bedd-945aba4e9f12 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:00.972850 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b299cae4-892d-47bf-bedd-945aba4e9f12 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:00.973128 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b299cae4-892d-47bf-bedd-945aba4e9f12 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:01.049408 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b299cae4-892d-47bf-bedd-945aba4e9f12 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:01.050911 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b299cae4-892d-47bf-bedd-945aba4e9f12 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:01.088385 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:01.160290 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 53/107] 10.222.0.33 () {58 vars in 1282 bytes} [Sun Oct 9 00:42:00 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 346 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 09 00:42:01.166123 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61d8310d-1a41-42e3-80d5-8e28f35f6b67 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:01.280296 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61d8310d-1a41-42e3-80d5-8e28f35f6b67 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:01.323175 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6068054e-8305-44bb-be43-fbde1486ea1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:01.323919 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6068054e-8305-44bb-be43-fbde1486ea1d None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:01.324210 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6068054e-8305-44bb-be43-fbde1486ea1d None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:01.324486 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6068054e-8305-44bb-be43-fbde1486ea1d None admin] PATH_INFO: `/v3/projects/768d646302f443f1a7d490b8c3d4bdbb` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:01.325648 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6068054e-8305-44bb-be43-fbde1486ea1d None admin] RBAC: Authorizing `identity:get_project(project_id=768d646302f443f1a7d490b8c3d4bdbb)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:01.327085 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6068054e-8305-44bb-be43-fbde1486ea1d None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:01.328748 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 55/108] 10.222.0.33 () {56 vars in 1186 bytes} [Sun Oct 9 00:42:01 2022] GET /identity/v3/projects/768d646302f443f1a7d490b8c3d4bdbb => generated 298 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:42:01.334788 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b299cae4-892d-47bf-bedd-945aba4e9f12 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:01.446064 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b299cae4-892d-47bf-bedd-945aba4e9f12 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:01.490390 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eecffa25-ba23-47bd-92f8-a286006ff8d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:01.491003 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eecffa25-ba23-47bd-92f8-a286006ff8d3 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:01.491286 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eecffa25-ba23-47bd-92f8-a286006ff8d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:01.491563 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eecffa25-ba23-47bd-92f8-a286006ff8d3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:01.492320 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eecffa25-ba23-47bd-92f8-a286006ff8d3 None admin] RBAC: Adding query filter params (role.id=2d631b6eec1646c9834f74efa1d66091, scope.project.id=768d646302f443f1a7d490b8c3d4bdbb, user.id=d90f753f3b2e4bb08eb926d2d604138e) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:01.492632 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eecffa25-ba23-47bd-92f8-a286006ff8d3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:01.494088 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eecffa25-ba23-47bd-92f8-a286006ff8d3 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:01.497391 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:01.502682 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 54/109] 10.222.0.33 () {56 vars in 1374 bytes} [Sun Oct 9 00:42:01 2022] GET /identity/v3/role_assignments?user.id=d90f753f3b2e4bb08eb926d2d604138e&scope.project.id=768d646302f443f1a7d490b8c3d4bdbb&role.id=2d631b6eec1646c9834f74efa1d66091 => generated 259 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:01.921637 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e1aff7-1ede-4b8b-b20b-217590c8c4e4 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:01.922341 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e1aff7-1ede-4b8b-b20b-217590c8c4e4 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:01.922341 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e1aff7-1ede-4b8b-b20b-217590c8c4e4 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:01.923851 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 56/110] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:42:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:42:01.929095 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea018896-8931-46ef-8ead-406df1b6916a None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:01.929911 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea018896-8931-46ef-8ead-406df1b6916a None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:01.930405 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea018896-8931-46ef-8ead-406df1b6916a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:01.957026 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-ea018896-8931-46ef-8ead-406df1b6916a None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:42:01.958707 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-ea018896-8931-46ef-8ead-406df1b6916a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:42:02.040404 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 55/111] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:42:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 09 00:42:02.134151 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-225bab79-956a-4cc3-a595-d6efc43a1048 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:02.134860 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-225bab79-956a-4cc3-a595-d6efc43a1048 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:02.135341 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-225bab79-956a-4cc3-a595-d6efc43a1048 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:02.137471 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 57/112] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:42:02 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:42:02.142842 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f09a5ba8-cfca-4958-b558-230a40c4d357 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:02.143414 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f09a5ba8-cfca-4958-b558-230a40c4d357 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:02.143903 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f09a5ba8-cfca-4958-b558-230a40c4d357 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:02.168618 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-f09a5ba8-cfca-4958-b558-230a40c4d357 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:42:02.170177 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-f09a5ba8-cfca-4958-b558-230a40c4d357 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:42:02.252431 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 56/113] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:42:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 09 00:42:02.291489 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f89e2d7a-1c53-4163-8861-87149d4a906e None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:02.292707 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f89e2d7a-1c53-4163-8861-87149d4a906e None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:02.293184 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f89e2d7a-1c53-4163-8861-87149d4a906e None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:02.294769 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 58/114] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:42:02 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:42:02.300272 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2216292e-957d-44fb-a2d7-497d2128ab12 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:02.300794 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2216292e-957d-44fb-a2d7-497d2128ab12 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:02.301301 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2216292e-957d-44fb-a2d7-497d2128ab12 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:02.305215 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2a34ce7-9e24-4e24-9d85-55e154d18397 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:02.305571 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2a34ce7-9e24-4e24-9d85-55e154d18397 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:02.305942 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2a34ce7-9e24-4e24-9d85-55e154d18397 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:02.307393 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 59/115] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:42:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 09 00:42:02.313543 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-927ab54e-3686-41d2-ac19-2f334e4cefea None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:02.313905 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-927ab54e-3686-41d2-ac19-2f334e4cefea None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:02.314304 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-927ab54e-3686-41d2-ac19-2f334e4cefea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:02.328628 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-2216292e-957d-44fb-a2d7-497d2128ab12 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:42:02.329835 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-2216292e-957d-44fb-a2d7-497d2128ab12 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:42:02.334495 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-927ab54e-3686-41d2-ac19-2f334e4cefea None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:42:02.335808 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-927ab54e-3686-41d2-ac19-2f334e4cefea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:42:02.356506 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:02.429127 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 60/116] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:42:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 09 00:42:02.443535 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:02.462191 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 57/117] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:42:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 09 00:42:02.491776 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-927ab54e-3686-41d2-ac19-2f334e4cefea None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:02.615467 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-927ab54e-3686-41d2-ac19-2f334e4cefea None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:02.660694 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f031329c-7a12-4d74-a754-cca028a50fb3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:02.661331 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f031329c-7a12-4d74-a754-cca028a50fb3 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:02.661614 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f031329c-7a12-4d74-a754-cca028a50fb3 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:02.661941 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f031329c-7a12-4d74-a754-cca028a50fb3 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:02.676796 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f031329c-7a12-4d74-a754-cca028a50fb3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:42:02.681922 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-f031329c-7a12-4d74-a754-cca028a50fb3 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 09 00:42:02.681922 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:42:02.681922 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:42:02.681922 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:42:02.681922 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:42:02.681922 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:42:02.681922 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:42:02.681922 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:42:02.681922 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:42:02.681922 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:42:02.681922 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:42:02.681922 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:42:02.681922 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:42:02.681922 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:42:02.681922 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:42:02.681922 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:42:02.683357 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:42:02.683357 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:42:02.683357 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:42:02.683357 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:42:02.683357 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:42:02.683357 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:42:02.683357 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:42:02.683357 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:42:02.683357 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:42:02.683357 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:42:02.683357 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:42:02.683357 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:42:02.683357 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:42:02.683357 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:42:02.683357 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:42:02.683357 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:42:02.684213 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:42:02.684213 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:42:02.684213 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:42:02.684213 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:42:02.684213 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:42:02.684213 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:42:02.684213 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:42:02.684213 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:42:02.684213 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:42:02.684213 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:42:02.684213 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 09 00:42:02.684213 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:42:02.684213 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 61/118] 10.222.0.33 () {56 vars in 1096 bytes} [Sun Oct 9 00:42:02 2022] GET /identity/v3/roles/admin => generated 83 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:42:02.691002 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2216292e-957d-44fb-a2d7-497d2128ab12 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:02.792848 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f031329c-7a12-4d74-a754-cca028a50fb3 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:02.817410 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:02.833918 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2216292e-957d-44fb-a2d7-497d2128ab12 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:02.912298 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a13a3533-2b83-4743-b714-aa21a598f62d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:02.912919 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13a3533-2b83-4743-b714-aa21a598f62d None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:02.913199 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13a3533-2b83-4743-b714-aa21a598f62d None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:02.913469 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13a3533-2b83-4743-b714-aa21a598f62d None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:02.914132 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a13a3533-2b83-4743-b714-aa21a598f62d None admin] RBAC: Adding query filter params (name=admin) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:02.914439 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a13a3533-2b83-4743-b714-aa21a598f62d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:02.916120 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a13a3533-2b83-4743-b714-aa21a598f62d None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:02.931239 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 58/119] 10.222.0.33 () {56 vars in 1099 bytes} [Sun Oct 9 00:42:02 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 241 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:02.937989 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a13a3533-2b83-4743-b714-aa21a598f62d None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:02.956106 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f031329c-7a12-4d74-a754-cca028a50fb3 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:02.999257 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cd50845-a047-4b48-9ffd-f0fbbce0a3b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:03.000219 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd50845-a047-4b48-9ffd-f0fbbce0a3b7 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:03.000642 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd50845-a047-4b48-9ffd-f0fbbce0a3b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:03.001052 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd50845-a047-4b48-9ffd-f0fbbce0a3b7 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:03.012624 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cd50845-a047-4b48-9ffd-f0fbbce0a3b7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:42:03.018765 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-5cd50845-a047-4b48-9ffd-f0fbbce0a3b7 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 09 00:42:03.018765 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:42:03.018765 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:42:03.018765 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:42:03.018765 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:42:03.018765 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:42:03.018765 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:42:03.018765 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:42:03.018765 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:42:03.018765 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:42:03.018765 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:42:03.018765 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:42:03.018765 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:42:03.018765 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:42:03.018765 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:42:03.018765 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:42:03.019833 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:42:03.019833 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:42:03.019833 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:42:03.019833 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:42:03.019833 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:42:03.019833 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:42:03.019833 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:42:03.019833 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:42:03.019833 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:42:03.019833 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:42:03.019833 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:42:03.019833 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:42:03.019833 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:42:03.019833 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:42:03.019833 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:42:03.019833 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:42:03.020776 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:42:03.020776 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:42:03.020776 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:42:03.020776 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:42:03.020776 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:42:03.020776 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:42:03.020776 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:42:03.020776 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:42:03.020776 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:42:03.020776 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:42:03.020776 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 09 00:42:03.020776 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:42:03.023157 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 62/120] 10.222.0.33 () {56 vars in 1099 bytes} [Sun Oct 9 00:42:02 2022] GET /identity/v3/roles/member => generated 84 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:42:03.024197 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cd50845-a047-4b48-9ffd-f0fbbce0a3b7 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:03.040996 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:03.097437 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a13a3533-2b83-4743-b714-aa21a598f62d None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:03.112233 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:03.133199 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e85058fa-2c3a-4ef4-9716-56ff77580c83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:03.133811 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e85058fa-2c3a-4ef4-9716-56ff77580c83 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:03.134091 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e85058fa-2c3a-4ef4-9716-56ff77580c83 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:03.134363 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e85058fa-2c3a-4ef4-9716-56ff77580c83 None admin] PATH_INFO: `/v3/users/d9c8218a78034540ba9ed09b697588a8` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:03.135760 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e85058fa-2c3a-4ef4-9716-56ff77580c83 None admin] RBAC: Authorizing `identity:get_user(user_id=d9c8218a78034540ba9ed09b697588a8)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:03.137795 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e85058fa-2c3a-4ef4-9716-56ff77580c83 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:03.139665 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 59/121] 10.222.0.33 () {56 vars in 1177 bytes} [Sun Oct 9 00:42:02 2022] GET /identity/v3/users/d9c8218a78034540ba9ed09b697588a8 => generated 247 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:03.140495 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e85058fa-2c3a-4ef4-9716-56ff77580c83 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:03.142223 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cd50845-a047-4b48-9ffd-f0fbbce0a3b7 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:03.180452 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5375c1c2-d8b3-4328-b9f8-c1cb3cbff97d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:03.181422 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5375c1c2-d8b3-4328-b9f8-c1cb3cbff97d None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:03.181901 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5375c1c2-d8b3-4328-b9f8-c1cb3cbff97d None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:03.182369 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5375c1c2-d8b3-4328-b9f8-c1cb3cbff97d None admin] PATH_INFO: `/v3/roles/member` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:03.194489 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5375c1c2-d8b3-4328-b9f8-c1cb3cbff97d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:42:03.201207 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-5375c1c2-d8b3-4328-b9f8-c1cb3cbff97d None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 09 00:42:03.201207 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:42:03.201207 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:42:03.201207 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:42:03.201207 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:42:03.201207 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:42:03.201207 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:42:03.201207 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:42:03.201207 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:42:03.201207 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:42:03.201207 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:42:03.201207 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:42:03.201207 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:42:03.201207 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:42:03.201207 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:42:03.201207 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:42:03.202025 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:42:03.202025 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:42:03.202025 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:42:03.202025 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:42:03.202025 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:42:03.202025 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:42:03.202025 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:42:03.202025 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:42:03.202025 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:42:03.202025 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:42:03.202025 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:42:03.202025 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:42:03.202025 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:42:03.202025 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:42:03.202025 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:42:03.202025 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:42:03.202766 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:42:03.202766 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:42:03.202766 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:42:03.202766 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:42:03.202766 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:42:03.202766 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:42:03.202766 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:42:03.202766 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:42:03.202766 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:42:03.202766 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:42:03.202766 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 09 00:42:03.202766 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:42:03.205137 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 63/122] 10.222.0.33 () {56 vars in 1099 bytes} [Sun Oct 9 00:42:03 2022] GET /identity/v3/roles/member => generated 84 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:42:03.206157 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5375c1c2-d8b3-4328-b9f8-c1cb3cbff97d None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:03.250159 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e85058fa-2c3a-4ef4-9716-56ff77580c83 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:03.270368 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:03.285795 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0479d03-e859-4ed1-a53c-64dfb81dafe1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:03.286418 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0479d03-e859-4ed1-a53c-64dfb81dafe1 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:03.286708 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0479d03-e859-4ed1-a53c-64dfb81dafe1 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:03.286977 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0479d03-e859-4ed1-a53c-64dfb81dafe1 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:03.297438 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0479d03-e859-4ed1-a53c-64dfb81dafe1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:42:03.302609 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-f0479d03-e859-4ed1-a53c-64dfb81dafe1 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 09 00:42:03.302609 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:42:03.302609 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:42:03.302609 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:42:03.302609 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:42:03.302609 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:42:03.302609 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:42:03.302609 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:42:03.302609 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:42:03.302609 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:42:03.302609 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:42:03.302609 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:42:03.302609 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:42:03.302609 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:42:03.302609 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:42:03.302609 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:42:03.303440 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:42:03.303440 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:42:03.303440 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:42:03.303440 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:42:03.303440 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:42:03.303440 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:42:03.303440 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:42:03.303440 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:42:03.303440 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:42:03.303440 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:42:03.303440 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:42:03.303440 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:42:03.303440 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:42:03.303440 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:42:03.303440 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:42:03.303440 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:42:03.304510 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:42:03.304510 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:42:03.304510 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:42:03.304510 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:42:03.304510 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:42:03.304510 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:42:03.304510 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:42:03.304510 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:42:03.304510 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:42:03.304510 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:42:03.304510 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 09 00:42:03.304510 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:42:03.304510 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 60/123] 10.222.0.33 () {56 vars in 1114 bytes} [Sun Oct 9 00:42:03 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:42:03.304510 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0479d03-e859-4ed1-a53c-64dfb81dafe1 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:03.331762 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5375c1c2-d8b3-4328-b9f8-c1cb3cbff97d None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:03.361578 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:03.367176 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b80721c-f171-4b8a-acb6-54de9f041e2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:03.368124 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b80721c-f171-4b8a-acb6-54de9f041e2f None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:03.368627 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b80721c-f171-4b8a-acb6-54de9f041e2f None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:03.369153 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b80721c-f171-4b8a-acb6-54de9f041e2f None admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:03.370000 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b80721c-f171-4b8a-acb6-54de9f041e2f None admin] RBAC: Adding query filter params (name=member) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:03.370838 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b80721c-f171-4b8a-acb6-54de9f041e2f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:03.372271 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b80721c-f171-4b8a-acb6-54de9f041e2f None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:03.380975 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 64/124] 10.222.0.33 () {56 vars in 1101 bytes} [Sun Oct 9 00:42:03 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:03.381843 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b80721c-f171-4b8a-acb6-54de9f041e2f None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:03.420077 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0479d03-e859-4ed1-a53c-64dfb81dafe1 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:03.457738 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d0c43aa-b6b2-4e42-ab52-7bfa82b6df5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:03.458875 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d0c43aa-b6b2-4e42-ab52-7bfa82b6df5e None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:03.459356 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d0c43aa-b6b2-4e42-ab52-7bfa82b6df5e None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:03.459861 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d0c43aa-b6b2-4e42-ab52-7bfa82b6df5e None admin] PATH_INFO: `/v3/projects/768d646302f443f1a7d490b8c3d4bdbb` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:03.461506 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d0c43aa-b6b2-4e42-ab52-7bfa82b6df5e None admin] RBAC: Authorizing `identity:get_project(project_id=768d646302f443f1a7d490b8c3d4bdbb)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:03.463279 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d0c43aa-b6b2-4e42-ab52-7bfa82b6df5e None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:03.465329 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 61/125] 10.222.0.33 () {56 vars in 1186 bytes} [Sun Oct 9 00:42:03 2022] GET /identity/v3/projects/768d646302f443f1a7d490b8c3d4bdbb => generated 298 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:03.466509 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d0c43aa-b6b2-4e42-ab52-7bfa82b6df5e None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:03.484353 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b80721c-f171-4b8a-acb6-54de9f041e2f None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:03.521007 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efceba48-42bc-45ed-8e18-000698e0f162 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:03.521625 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efceba48-42bc-45ed-8e18-000698e0f162 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:03.521910 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efceba48-42bc-45ed-8e18-000698e0f162 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:03.522232 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efceba48-42bc-45ed-8e18-000698e0f162 None admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:03.522898 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efceba48-42bc-45ed-8e18-000698e0f162 None admin] RBAC: Adding query filter params (name=member) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:03.523203 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efceba48-42bc-45ed-8e18-000698e0f162 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:03.523672 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:03.524650 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efceba48-42bc-45ed-8e18-000698e0f162 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:03.533585 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 65/126] 10.222.0.33 () {56 vars in 1101 bytes} [Sun Oct 9 00:42:03 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:03.534451 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efceba48-42bc-45ed-8e18-000698e0f162 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:03.585008 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d0c43aa-b6b2-4e42-ab52-7bfa82b6df5e None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:03.596914 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:03.631615 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3853e7b4-4d40-48fb-b694-d772937a4b54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:03.631615 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3853e7b4-4d40-48fb-b694-d772937a4b54 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:03.631615 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3853e7b4-4d40-48fb-b694-d772937a4b54 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:03.631615 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3853e7b4-4d40-48fb-b694-d772937a4b54 None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:03.632003 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3853e7b4-4d40-48fb-b694-d772937a4b54 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:03.632003 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3853e7b4-4d40-48fb-b694-d772937a4b54 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:03.632454 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3853e7b4-4d40-48fb-b694-d772937a4b54 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:03.641062 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 62/127] 10.222.0.33 () {56 vars in 1111 bytes} [Sun Oct 9 00:42:03 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:03.641941 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3853e7b4-4d40-48fb-b694-d772937a4b54 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:03.643090 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efceba48-42bc-45ed-8e18-000698e0f162 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:03.678955 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-446e3fc3-7f1d-453b-adf7-09efa85db6e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:03.679668 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-446e3fc3-7f1d-453b-adf7-09efa85db6e4 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:03.679955 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-446e3fc3-7f1d-453b-adf7-09efa85db6e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:03.680231 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-446e3fc3-7f1d-453b-adf7-09efa85db6e4 None admin] PATH_INFO: `/v3/users/da5fb3b5405648df8a2114ff2dca2d02` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:03.681425 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-446e3fc3-7f1d-453b-adf7-09efa85db6e4 None admin] RBAC: Authorizing `identity:get_user(user_id=da5fb3b5405648df8a2114ff2dca2d02)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:03.683431 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-446e3fc3-7f1d-453b-adf7-09efa85db6e4 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:03.685190 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 66/128] 10.222.0.33 () {56 vars in 1177 bytes} [Sun Oct 9 00:42:03 2022] GET /identity/v3/users/da5fb3b5405648df8a2114ff2dca2d02 => generated 275 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:03.685958 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-446e3fc3-7f1d-453b-adf7-09efa85db6e4 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:03.749117 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3853e7b4-4d40-48fb-b694-d772937a4b54 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:03.760383 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:03.791225 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-446e3fc3-7f1d-453b-adf7-09efa85db6e4 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:03.796006 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95d453a8-57fa-4124-92fa-541783d0ae3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:03.796728 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95d453a8-57fa-4124-92fa-541783d0ae3e None admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:03.797014 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95d453a8-57fa-4124-92fa-541783d0ae3e None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:03.797291 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95d453a8-57fa-4124-92fa-541783d0ae3e None admin] PATH_INFO: `/v3/projects/768d646302f443f1a7d490b8c3d4bdbb/users/d9c8218a78034540ba9ed09b697588a8/roles/092036ee46ac427baf4d56468575f697` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:03.799753 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95d453a8-57fa-4124-92fa-541783d0ae3e None admin] RBAC: Authorizing `identity:create_grant(project_id=768d646302f443f1a7d490b8c3d4bdbb, user_id=d9c8218a78034540ba9ed09b697588a8, role_id=092036ee46ac427baf4d56468575f697)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:03.802101 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95d453a8-57fa-4124-92fa-541783d0ae3e None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:03.816895 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:03.820025 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 63/129] 10.222.0.33 () {58 vars in 1439 bytes} [Sun Oct 9 00:42:03 2022] PUT /identity/v3/projects/768d646302f443f1a7d490b8c3d4bdbb/users/d9c8218a78034540ba9ed09b697588a8/roles/092036ee46ac427baf4d56468575f697 => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:42:03.822028 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95d453a8-57fa-4124-92fa-541783d0ae3e None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:03.831069 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-400c16c0-77c9-465a-a726-9174e864926d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:03.831964 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-400c16c0-77c9-465a-a726-9174e864926d None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:03.832310 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-400c16c0-77c9-465a-a726-9174e864926d None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:03.832655 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-400c16c0-77c9-465a-a726-9174e864926d None admin] PATH_INFO: `/v3/users/da5fb3b5405648df8a2114ff2dca2d02` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:03.834716 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-400c16c0-77c9-465a-a726-9174e864926d None admin] RBAC: Authorizing `identity:get_user(user_id=da5fb3b5405648df8a2114ff2dca2d02)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:03.836314 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-400c16c0-77c9-465a-a726-9174e864926d None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:03.838673 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 67/130] 10.222.0.33 () {56 vars in 1177 bytes} [Sun Oct 9 00:42:03 2022] GET /identity/v3/users/da5fb3b5405648df8a2114ff2dca2d02 => generated 275 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:03.840298 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-400c16c0-77c9-465a-a726-9174e864926d None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:03.993416 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95d453a8-57fa-4124-92fa-541783d0ae3e None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:04.006088 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-400c16c0-77c9-465a-a726-9174e864926d None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:04.038211 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4b90d29-86e2-4de3-a3c3-e905a747b0ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:04.039225 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4b90d29-86e2-4de3-a3c3-e905a747b0ec None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:04.040004 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4b90d29-86e2-4de3-a3c3-e905a747b0ec None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:04.040455 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4b90d29-86e2-4de3-a3c3-e905a747b0ec None admin] PATH_INFO: `/v3/users/da5fb3b5405648df8a2114ff2dca2d02` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:04.041898 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4b90d29-86e2-4de3-a3c3-e905a747b0ec None admin] RBAC: Authorizing `identity:get_user(user_id=da5fb3b5405648df8a2114ff2dca2d02)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:04.043699 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4b90d29-86e2-4de3-a3c3-e905a747b0ec None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:04.045568 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 64/131] 10.222.0.33 () {56 vars in 1177 bytes} [Sun Oct 9 00:42:03 2022] GET /identity/v3/users/da5fb3b5405648df8a2114ff2dca2d02 => generated 275 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:04.046633 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4b90d29-86e2-4de3-a3c3-e905a747b0ec None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:04.062512 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8313ce72-95c3-4d1f-8507-9540d966f536 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:04.063156 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8313ce72-95c3-4d1f-8507-9540d966f536 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:04.063501 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8313ce72-95c3-4d1f-8507-9540d966f536 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:04.063811 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8313ce72-95c3-4d1f-8507-9540d966f536 None admin] PATH_INFO: `/v3/projects/fc09cf0f89194e89b47f7e4e63ad25da` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:04.065743 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8313ce72-95c3-4d1f-8507-9540d966f536 None admin] RBAC: Authorizing `identity:get_project(project_id=fc09cf0f89194e89b47f7e4e63ad25da)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:04.067550 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8313ce72-95c3-4d1f-8507-9540d966f536 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:04.070038 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 68/132] 10.222.0.33 () {56 vars in 1186 bytes} [Sun Oct 9 00:42:03 2022] GET /identity/v3/projects/fc09cf0f89194e89b47f7e4e63ad25da => generated 312 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:04.070647 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8313ce72-95c3-4d1f-8507-9540d966f536 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:04.102187 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:04.139947 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:04.189713 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8313ce72-95c3-4d1f-8507-9540d966f536 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:04.194943 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4b90d29-86e2-4de3-a3c3-e905a747b0ec None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:04.227821 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-195ff2f1-eaca-4ebb-a00f-06e8eb01e37d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:04.229285 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-195ff2f1-eaca-4ebb-a00f-06e8eb01e37d None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:04.230246 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-195ff2f1-eaca-4ebb-a00f-06e8eb01e37d None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:04.230662 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-195ff2f1-eaca-4ebb-a00f-06e8eb01e37d None admin] PATH_INFO: `/v3/projects/768d646302f443f1a7d490b8c3d4bdbb` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:04.231962 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-195ff2f1-eaca-4ebb-a00f-06e8eb01e37d None admin] RBAC: Authorizing `identity:get_project(project_id=768d646302f443f1a7d490b8c3d4bdbb)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:04.233931 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-195ff2f1-eaca-4ebb-a00f-06e8eb01e37d None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:04.236248 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 69/133] 10.222.0.33 () {56 vars in 1186 bytes} [Sun Oct 9 00:42:04 2022] GET /identity/v3/projects/768d646302f443f1a7d490b8c3d4bdbb => generated 298 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:04.236555 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ea3b1d1-4b0a-4aea-a549-b3542b785983 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:04.237305 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-195ff2f1-eaca-4ebb-a00f-06e8eb01e37d None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:04.237416 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ea3b1d1-4b0a-4aea-a549-b3542b785983 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:04.237622 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ea3b1d1-4b0a-4aea-a549-b3542b785983 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:04.237902 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ea3b1d1-4b0a-4aea-a549-b3542b785983 None admin] PATH_INFO: `/v3/projects/768d646302f443f1a7d490b8c3d4bdbb` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:04.239043 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ea3b1d1-4b0a-4aea-a549-b3542b785983 None admin] RBAC: Authorizing `identity:get_project(project_id=768d646302f443f1a7d490b8c3d4bdbb)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:04.240875 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ea3b1d1-4b0a-4aea-a549-b3542b785983 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:04.243686 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 65/134] 10.222.0.33 () {56 vars in 1186 bytes} [Sun Oct 9 00:42:04 2022] GET /identity/v3/projects/768d646302f443f1a7d490b8c3d4bdbb => generated 298 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:04.244851 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79319d48-3299-40e1-93bf-b6f6b2c25378 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:04.246315 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79319d48-3299-40e1-93bf-b6f6b2c25378 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:04.246603 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79319d48-3299-40e1-93bf-b6f6b2c25378 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:04.247992 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 66/135] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:42:04 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:42:04.252345 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79319d48-3299-40e1-93bf-b6f6b2c25378 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:04.367677 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-195ff2f1-eaca-4ebb-a00f-06e8eb01e37d None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:04.398867 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:04.401747 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79319d48-3299-40e1-93bf-b6f6b2c25378 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:04.408318 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ea13d7d-7300-4682-a882-1948d58332f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:04.409210 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea13d7d-7300-4682-a882-1948d58332f5 None admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:04.409709 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea13d7d-7300-4682-a882-1948d58332f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:04.410181 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea13d7d-7300-4682-a882-1948d58332f5 None admin] PATH_INFO: `/v3/projects/fc09cf0f89194e89b47f7e4e63ad25da/users/da5fb3b5405648df8a2114ff2dca2d02/roles/588016933bab436d8b7c3ac76827356d` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:04.411410 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:04.412511 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ea13d7d-7300-4682-a882-1948d58332f5 None admin] RBAC: Authorizing `identity:create_grant(project_id=fc09cf0f89194e89b47f7e4e63ad25da, user_id=da5fb3b5405648df8a2114ff2dca2d02, role_id=588016933bab436d8b7c3ac76827356d)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:04.414184 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ea13d7d-7300-4682-a882-1948d58332f5 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:04.434339 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 70/136] 10.222.0.33 () {58 vars in 1439 bytes} [Sun Oct 9 00:42:04 2022] PUT /identity/v3/projects/fc09cf0f89194e89b47f7e4e63ad25da/users/da5fb3b5405648df8a2114ff2dca2d02/roles/588016933bab436d8b7c3ac76827356d => generated 0 bytes in 197 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:42:04.435941 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ea13d7d-7300-4682-a882-1948d58332f5 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:04.451169 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72e7a6c5-c6cf-402c-b676-f31ab69c1434 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:04.452033 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72e7a6c5-c6cf-402c-b676-f31ab69c1434 None admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:04.452391 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72e7a6c5-c6cf-402c-b676-f31ab69c1434 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:04.453362 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72e7a6c5-c6cf-402c-b676-f31ab69c1434 None admin] PATH_INFO: `/v3/projects/768d646302f443f1a7d490b8c3d4bdbb/users/da5fb3b5405648df8a2114ff2dca2d02/roles/7c1fdac272a34be99661077b0f1b8014` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:04.456693 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72e7a6c5-c6cf-402c-b676-f31ab69c1434 None admin] RBAC: Authorizing `identity:create_grant(project_id=768d646302f443f1a7d490b8c3d4bdbb, user_id=da5fb3b5405648df8a2114ff2dca2d02, role_id=7c1fdac272a34be99661077b0f1b8014)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:04.458593 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72e7a6c5-c6cf-402c-b676-f31ab69c1434 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:04.480470 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 67/137] 10.222.0.33 () {58 vars in 1439 bytes} [Sun Oct 9 00:42:04 2022] PUT /identity/v3/projects/768d646302f443f1a7d490b8c3d4bdbb/users/da5fb3b5405648df8a2114ff2dca2d02/roles/7c1fdac272a34be99661077b0f1b8014 => generated 0 bytes in 232 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:42:04.482742 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a585d0b-5040-43d3-bdd4-38d4d0d3f741 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:04.483292 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a585d0b-5040-43d3-bdd4-38d4d0d3f741 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:04.483759 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a585d0b-5040-43d3-bdd4-38d4d0d3f741 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:04.510343 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-9a585d0b-5040-43d3-bdd4-38d4d0d3f741 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:42:04.513137 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-9a585d0b-5040-43d3-bdd4-38d4d0d3f741 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:42:04.570795 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ea13d7d-7300-4682-a882-1948d58332f5 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:04.620682 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 68/138] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:42:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 09 00:42:04.622306 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a045e729-4e07-4214-a224-66f9937a8b8e None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:04.622954 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a045e729-4e07-4214-a224-66f9937a8b8e None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:04.623309 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a045e729-4e07-4214-a224-66f9937a8b8e None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:04.627337 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 69/139] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:42:04 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:42:04.633037 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b46436cf-8beb-4c21-bf44-bc279d7f8446 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:04.633315 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b46436cf-8beb-4c21-bf44-bc279d7f8446 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:04.633801 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b46436cf-8beb-4c21-bf44-bc279d7f8446 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:04.661411 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cb6ad69-7b3f-4438-ac32-6738a36ca004 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:04.662482 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb6ad69-7b3f-4438-ac32-6738a36ca004 None admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:04.662943 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb6ad69-7b3f-4438-ac32-6738a36ca004 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:04.663423 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb6ad69-7b3f-4438-ac32-6738a36ca004 None admin] PATH_INFO: `/v3/projects/768d646302f443f1a7d490b8c3d4bdbb/users/da5fb3b5405648df8a2114ff2dca2d02/roles/588016933bab436d8b7c3ac76827356d` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:04.667620 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-b46436cf-8beb-4c21-bf44-bc279d7f8446 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:42:04.667620 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-b46436cf-8beb-4c21-bf44-bc279d7f8446 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:42:04.672862 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cb6ad69-7b3f-4438-ac32-6738a36ca004 None admin] RBAC: Authorizing `identity:create_grant(project_id=768d646302f443f1a7d490b8c3d4bdbb, user_id=da5fb3b5405648df8a2114ff2dca2d02, role_id=588016933bab436d8b7c3ac76827356d)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:04.674710 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cb6ad69-7b3f-4438-ac32-6738a36ca004 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:04.716376 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 71/140] 10.222.0.33 () {58 vars in 1439 bytes} [Sun Oct 9 00:42:04 2022] PUT /identity/v3/projects/768d646302f443f1a7d490b8c3d4bdbb/users/da5fb3b5405648df8a2114ff2dca2d02/roles/588016933bab436d8b7c3ac76827356d => generated 0 bytes in 281 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:42:04.767819 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 70/141] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:42:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 09 00:42:05.075566 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cb6ad69-7b3f-4438-ac32-6738a36ca004 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:05.129493 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b46436cf-8beb-4c21-bf44-bc279d7f8446 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:05.130643 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:05.182117 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:05.221791 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cb6ad69-7b3f-4438-ac32-6738a36ca004 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:05.301226 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-676f3003-4afd-4016-8cb1-f11b234ecf9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:05.301882 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676f3003-4afd-4016-8cb1-f11b234ecf9e None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:05.302168 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676f3003-4afd-4016-8cb1-f11b234ecf9e None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:05.302446 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676f3003-4afd-4016-8cb1-f11b234ecf9e None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:05.325805 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-676f3003-4afd-4016-8cb1-f11b234ecf9e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:42:05.336599 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-676f3003-4afd-4016-8cb1-f11b234ecf9e None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 09 00:42:05.336599 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:42:05.336599 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:42:05.336599 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:42:05.336599 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:42:05.336599 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:42:05.336599 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:42:05.336599 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:42:05.336599 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:42:05.336599 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:42:05.336599 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:42:05.336599 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:42:05.336599 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:42:05.336599 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:42:05.336599 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:42:05.336599 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:42:05.337451 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:42:05.337451 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:42:05.337451 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:42:05.337451 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:42:05.337451 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:42:05.337451 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:42:05.337451 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:42:05.337451 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:42:05.337451 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:42:05.337451 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:42:05.337451 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:42:05.337451 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:42:05.337451 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:42:05.337451 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:42:05.337451 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:42:05.337451 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:42:05.338176 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:42:05.338176 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:42:05.338176 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:42:05.338176 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:42:05.338176 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:42:05.338176 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:42:05.338176 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:42:05.338176 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:42:05.338176 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:42:05.338176 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:42:05.338176 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 09 00:42:05.338176 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:42:05.338176 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 72/142] 10.222.0.33 () {56 vars in 1099 bytes} [Sun Oct 9 00:42:05 2022] GET /identity/v3/roles/reader => generated 84 bytes in 263 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:42:05.342590 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b46436cf-8beb-4c21-bf44-bc279d7f8446 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:05.345477 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-676f3003-4afd-4016-8cb1-f11b234ecf9e None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:05.440354 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25593e74-1642-438e-9315-656e67c6856a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:05.441011 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25593e74-1642-438e-9315-656e67c6856a None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:05.441311 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25593e74-1642-438e-9315-656e67c6856a None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:05.441557 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25593e74-1642-438e-9315-656e67c6856a None admin] PATH_INFO: `/v3/domains/default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:05.448872 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25593e74-1642-438e-9315-656e67c6856a None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:05.450373 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25593e74-1642-438e-9315-656e67c6856a None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:05.452016 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 71/143] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:42:05 2022] GET /identity/v3/domains/default => generated 199 bytes in 323 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:05.457788 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25593e74-1642-438e-9315-656e67c6856a None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:05.505840 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-676f3003-4afd-4016-8cb1-f11b234ecf9e None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:05.521849 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:05.597485 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9705dc5-b1b5-4286-8649-8894f2328927 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:05.598031 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9705dc5-b1b5-4286-8649-8894f2328927 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:05.598315 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9705dc5-b1b5-4286-8649-8894f2328927 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:05.598585 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9705dc5-b1b5-4286-8649-8894f2328927 None admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:05.599252 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9705dc5-b1b5-4286-8649-8894f2328927 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:05.599553 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9705dc5-b1b5-4286-8649-8894f2328927 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:05.601081 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9705dc5-b1b5-4286-8649-8894f2328927 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:05.610624 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 73/144] 10.222.0.33 () {56 vars in 1101 bytes} [Sun Oct 9 00:42:05 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 265 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:05.616123 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9705dc5-b1b5-4286-8649-8894f2328927 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:05.688200 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:05.700740 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25593e74-1642-438e-9315-656e67c6856a None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:05.757131 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9705dc5-b1b5-4286-8649-8894f2328927 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:05.788894 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ef09b61-c9c5-4d6a-bf0c-f3e809a7ecb6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:05.789781 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ef09b61-c9c5-4d6a-bf0c-f3e809a7ecb6 None admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:05.790061 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ef09b61-c9c5-4d6a-bf0c-f3e809a7ecb6 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:05.790337 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ef09b61-c9c5-4d6a-bf0c-f3e809a7ecb6 None admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:05.791001 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ef09b61-c9c5-4d6a-bf0c-f3e809a7ecb6 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:05.793047 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ef09b61-c9c5-4d6a-bf0c-f3e809a7ecb6 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:05.809681 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d6bfb90-8e7c-419c-a2b9-376c175c35cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:05.810325 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d6bfb90-8e7c-419c-a2b9-376c175c35cb None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:05.810640 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d6bfb90-8e7c-419c-a2b9-376c175c35cb None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:05.810890 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d6bfb90-8e7c-419c-a2b9-376c175c35cb None admin] PATH_INFO: `/v3/users/d90f753f3b2e4bb08eb926d2d604138e` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:05.816293 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d6bfb90-8e7c-419c-a2b9-376c175c35cb None admin] RBAC: Authorizing `identity:get_user(user_id=d90f753f3b2e4bb08eb926d2d604138e)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:05.817757 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d6bfb90-8e7c-419c-a2b9-376c175c35cb None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:05.819372 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 74/145] 10.222.0.33 () {56 vars in 1177 bytes} [Sun Oct 9 00:42:05 2022] GET /identity/v3/users/d90f753f3b2e4bb08eb926d2d604138e => generated 289 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:42:05.824924 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d6bfb90-8e7c-419c-a2b9-376c175c35cb None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:05.844265 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 72/146] 10.222.0.33 () {60 vars in 1132 bytes} [Sun Oct 9 00:42:05 2022] POST /identity/v3/users => generated 283 bytes in 387 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:42:05.893272 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:05.980693 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d6bfb90-8e7c-419c-a2b9-376c175c35cb None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:06.027687 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a65402f6-45b4-48b7-a0f5-974d4cb93e92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:06.028354 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a65402f6-45b4-48b7-a0f5-974d4cb93e92 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:06.028616 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a65402f6-45b4-48b7-a0f5-974d4cb93e92 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:06.028884 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a65402f6-45b4-48b7-a0f5-974d4cb93e92 None admin] PATH_INFO: `/v3/projects/768d646302f443f1a7d490b8c3d4bdbb` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:06.030059 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a65402f6-45b4-48b7-a0f5-974d4cb93e92 None admin] RBAC: Authorizing `identity:get_project(project_id=768d646302f443f1a7d490b8c3d4bdbb)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:06.031521 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a65402f6-45b4-48b7-a0f5-974d4cb93e92 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:06.033207 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 75/147] 10.222.0.33 () {56 vars in 1186 bytes} [Sun Oct 9 00:42:05 2022] GET /identity/v3/projects/768d646302f443f1a7d490b8c3d4bdbb => generated 298 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:06.040652 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ef09b61-c9c5-4d6a-bf0c-f3e809a7ecb6 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:06.220141 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ef09b61-c9c5-4d6a-bf0c-f3e809a7ecb6 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:06.256943 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c403e850-263d-4057-a5e2-8116775fe792 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:06.257608 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c403e850-263d-4057-a5e2-8116775fe792 None admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:06.257893 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c403e850-263d-4057-a5e2-8116775fe792 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:06.258174 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c403e850-263d-4057-a5e2-8116775fe792 None admin] PATH_INFO: `/v3/projects/768d646302f443f1a7d490b8c3d4bdbb/users/d90f753f3b2e4bb08eb926d2d604138e/roles/2d631b6eec1646c9834f74efa1d66091` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:06.260147 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c403e850-263d-4057-a5e2-8116775fe792 None admin] RBAC: Authorizing `identity:create_grant(project_id=768d646302f443f1a7d490b8c3d4bdbb, user_id=d90f753f3b2e4bb08eb926d2d604138e, role_id=2d631b6eec1646c9834f74efa1d66091)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:06.262180 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c403e850-263d-4057-a5e2-8116775fe792 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:06.287150 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 73/148] 10.222.0.33 () {58 vars in 1439 bytes} [Sun Oct 9 00:42:06 2022] PUT /identity/v3/projects/768d646302f443f1a7d490b8c3d4bdbb/users/d90f753f3b2e4bb08eb926d2d604138e/roles/2d631b6eec1646c9834f74efa1d66091 => generated 0 bytes in 247 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:42:06.730155 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c74278ae-b759-4a22-9655-cc2adefbeb87 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:06.730656 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c74278ae-b759-4a22-9655-cc2adefbeb87 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:06.730753 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c74278ae-b759-4a22-9655-cc2adefbeb87 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:06.732293 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 76/149] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:42:06 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:42:06.739263 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b29dc37e-3434-4f38-a1c9-b0822a464297 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:06.739556 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b29dc37e-3434-4f38-a1c9-b0822a464297 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:06.739862 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b29dc37e-3434-4f38-a1c9-b0822a464297 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:06.763978 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-b29dc37e-3434-4f38-a1c9-b0822a464297 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:42:06.765172 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-b29dc37e-3434-4f38-a1c9-b0822a464297 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:42:06.768711 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:06.859745 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 74/150] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:42:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 09 00:42:07.414014 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c74278ae-b759-4a22-9655-cc2adefbeb87 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:07.494450 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-341404f8-6bda-43dc-91d0-7540d1c10d67 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:07.495030 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-341404f8-6bda-43dc-91d0-7540d1c10d67 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:07.495030 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-341404f8-6bda-43dc-91d0-7540d1c10d67 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:07.496373 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 75/151] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:42:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:42:07.501711 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af325a7-ed54-4cf3-8071-0f2b45a1c940 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:07.501980 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af325a7-ed54-4cf3-8071-0f2b45a1c940 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:07.502250 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af325a7-ed54-4cf3-8071-0f2b45a1c940 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:07.535252 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-6af325a7-ed54-4cf3-8071-0f2b45a1c940 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:42:07.536458 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-6af325a7-ed54-4cf3-8071-0f2b45a1c940 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:42:07.588033 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c74278ae-b759-4a22-9655-cc2adefbeb87 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:07.598271 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:07.661281 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-992ea33a-8a23-49af-ada0-a0efe974466a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:07.661929 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992ea33a-8a23-49af-ada0-a0efe974466a None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:07.662208 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992ea33a-8a23-49af-ada0-a0efe974466a None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:07.662480 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992ea33a-8a23-49af-ada0-a0efe974466a None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:07.679259 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 76/152] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:42:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 09 00:42:07.681263 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-992ea33a-8a23-49af-ada0-a0efe974466a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:42:07.686704 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-992ea33a-8a23-49af-ada0-a0efe974466a None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 09 00:42:07.686704 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:42:07.686704 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:42:07.686704 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:42:07.686704 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:42:07.686704 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:42:07.686704 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:42:07.686704 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:42:07.686704 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:42:07.686704 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:42:07.686704 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:42:07.686704 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:42:07.686704 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:42:07.686704 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:42:07.686704 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:42:07.686704 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:42:07.687657 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:42:07.687657 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:42:07.687657 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:42:07.687657 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:42:07.687657 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:42:07.687657 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:42:07.687657 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:42:07.687657 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:42:07.687657 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:42:07.687657 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:42:07.687657 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:42:07.687657 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:42:07.687657 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:42:07.687657 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:42:07.687657 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:42:07.687657 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:42:07.688441 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:42:07.688441 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:42:07.688441 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:42:07.688441 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:42:07.688441 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:42:07.688441 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:42:07.688441 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:42:07.688441 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:42:07.688441 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:42:07.688441 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:42:07.688441 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 09 00:42:07.688441 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:42:07.692218 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 77/153] 10.222.0.33 () {56 vars in 1096 bytes} [Sun Oct 9 00:42:07 2022] GET /identity/v3/roles/admin => generated 83 bytes in 279 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:42:07.700276 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6af325a7-ed54-4cf3-8071-0f2b45a1c940 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:07.784714 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bfc3f80-2128-4fcf-9037-8e68fb3072ed None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:07.785260 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bfc3f80-2128-4fcf-9037-8e68fb3072ed None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:07.785260 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bfc3f80-2128-4fcf-9037-8e68fb3072ed None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:07.786786 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 78/154] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:42:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:42:07.794163 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c8f8f2-c962-499c-ba93-088ef18bf18d None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:07.794433 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c8f8f2-c962-499c-ba93-088ef18bf18d None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:07.794702 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c8f8f2-c962-499c-ba93-088ef18bf18d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:07.820844 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:07.824573 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-a9c8f8f2-c962-499c-ba93-088ef18bf18d None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:42:07.825773 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-a9c8f8f2-c962-499c-ba93-088ef18bf18d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:42:07.851349 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6af325a7-ed54-4cf3-8071-0f2b45a1c940 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:07.898477 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65479866-462f-416e-b342-faca87f46321 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:07.899103 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65479866-462f-416e-b342-faca87f46321 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:07.899381 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65479866-462f-416e-b342-faca87f46321 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:07.899890 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65479866-462f-416e-b342-faca87f46321 None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:07.900642 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65479866-462f-416e-b342-faca87f46321 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:07.900953 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65479866-462f-416e-b342-faca87f46321 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:07.902514 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65479866-462f-416e-b342-faca87f46321 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:07.919624 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 77/155] 10.222.0.33 () {56 vars in 1099 bytes} [Sun Oct 9 00:42:07 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:07.922220 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65479866-462f-416e-b342-faca87f46321 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:07.997342 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 79/156] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:42:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 204 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 09 00:42:08.077474 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65479866-462f-416e-b342-faca87f46321 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:08.116420 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:08.125469 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7366ba59-6929-4e44-9cf4-76678e4d3eab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:08.126722 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7366ba59-6929-4e44-9cf4-76678e4d3eab None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:08.127368 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7366ba59-6929-4e44-9cf4-76678e4d3eab None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:08.127659 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7366ba59-6929-4e44-9cf4-76678e4d3eab None admin] PATH_INFO: `/v3/users/d9c8218a78034540ba9ed09b697588a8` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:08.128806 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7366ba59-6929-4e44-9cf4-76678e4d3eab None admin] RBAC: Authorizing `identity:get_user(user_id=d9c8218a78034540ba9ed09b697588a8)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:08.130247 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7366ba59-6929-4e44-9cf4-76678e4d3eab None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:08.131835 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 78/157] 10.222.0.33 () {56 vars in 1177 bytes} [Sun Oct 9 00:42:07 2022] GET /identity/v3/users/d9c8218a78034540ba9ed09b697588a8 => generated 247 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:08.137767 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9c8f8f2-c962-499c-ba93-088ef18bf18d None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:08.172450 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ca2f4b-5181-47a1-866d-b3498cf28a17 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:08.172760 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ca2f4b-5181-47a1-866d-b3498cf28a17 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:08.173003 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ca2f4b-5181-47a1-866d-b3498cf28a17 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:08.174490 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 79/158] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:42:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 09 00:42:08.209630 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2fa7023-d0a6-4b45-b7e3-f8df23aa102e None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:08.209946 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2fa7023-d0a6-4b45-b7e3-f8df23aa102e None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:08.210172 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2fa7023-d0a6-4b45-b7e3-f8df23aa102e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:08.239938 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-b2fa7023-d0a6-4b45-b7e3-f8df23aa102e None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:42:08.241176 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-b2fa7023-d0a6-4b45-b7e3-f8df23aa102e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:42:08.247211 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:08.252909 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9c8f8f2-c962-499c-ba93-088ef18bf18d None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:08.290423 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b25ec20f-9502-4762-9f59-a11ffe27c3e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:08.291038 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25ec20f-9502-4762-9f59-a11ffe27c3e3 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:08.291540 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25ec20f-9502-4762-9f59-a11ffe27c3e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:08.292399 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25ec20f-9502-4762-9f59-a11ffe27c3e3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:08.368456 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b25ec20f-9502-4762-9f59-a11ffe27c3e3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:08.370158 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b25ec20f-9502-4762-9f59-a11ffe27c3e3 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:08.397717 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 80/159] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:42:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 09 00:42:08.492992 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:08.495911 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 80/160] 10.222.0.33 () {58 vars in 1282 bytes} [Sun Oct 9 00:42:08 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 358 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 09 00:42:08.496221 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2fa7023-d0a6-4b45-b7e3-f8df23aa102e None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:08.501527 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b25ec20f-9502-4762-9f59-a11ffe27c3e3 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:08.644784 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b25ec20f-9502-4762-9f59-a11ffe27c3e3 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:08.694474 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62e04d93-8ab2-4879-8c4f-9c1210da5f20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:08.695621 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e04d93-8ab2-4879-8c4f-9c1210da5f20 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:08.696101 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e04d93-8ab2-4879-8c4f-9c1210da5f20 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:08.696578 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e04d93-8ab2-4879-8c4f-9c1210da5f20 None admin] PATH_INFO: `/v3/projects/768d646302f443f1a7d490b8c3d4bdbb` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:08.698045 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62e04d93-8ab2-4879-8c4f-9c1210da5f20 None admin] RBAC: Authorizing `identity:get_project(project_id=768d646302f443f1a7d490b8c3d4bdbb)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:08.699805 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62e04d93-8ab2-4879-8c4f-9c1210da5f20 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:08.701677 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 81/161] 10.222.0.33 () {56 vars in 1186 bytes} [Sun Oct 9 00:42:08 2022] GET /identity/v3/projects/768d646302f443f1a7d490b8c3d4bdbb => generated 298 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:42:08.707408 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62e04d93-8ab2-4879-8c4f-9c1210da5f20 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:08.721873 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2fa7023-d0a6-4b45-b7e3-f8df23aa102e None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:08.785291 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:08.796562 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:08.799251 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aab2d6c-c16f-4109-b840-42d23c0f02e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:08.800008 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aab2d6c-c16f-4109-b840-42d23c0f02e3 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:08.800284 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aab2d6c-c16f-4109-b840-42d23c0f02e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:08.800553 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aab2d6c-c16f-4109-b840-42d23c0f02e3 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:08.828280 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aab2d6c-c16f-4109-b840-42d23c0f02e3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:42:08.831424 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-3aab2d6c-c16f-4109-b840-42d23c0f02e3 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 09 00:42:08.831424 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:42:08.831424 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:42:08.831424 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:42:08.831424 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:42:08.831424 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:42:08.831424 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:42:08.831424 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:42:08.831424 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:42:08.831424 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:42:08.831424 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:42:08.831424 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:42:08.831424 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:42:08.831424 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:42:08.831424 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:42:08.831424 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:42:08.832326 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:42:08.832326 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:42:08.832326 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:42:08.832326 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:42:08.832326 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:42:08.832326 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:42:08.832326 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:42:08.832326 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:42:08.832326 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:42:08.832326 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:42:08.832326 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:42:08.832326 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:42:08.832326 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:42:08.832326 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:42:08.832326 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:42:08.832326 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:42:08.833126 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:42:08.833126 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:42:08.833126 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:42:08.833126 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:42:08.833126 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:42:08.833126 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:42:08.833126 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:42:08.833126 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:42:08.833126 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:42:08.833126 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:42:08.833126 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 09 00:42:08.833126 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:42:08.833126 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 81/162] 10.222.0.33 () {56 vars in 1099 bytes} [Sun Oct 9 00:42:08 2022] GET /identity/v3/roles/member => generated 84 bytes in 337 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 09 00:42:08.833803 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aab2d6c-c16f-4109-b840-42d23c0f02e3 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:08.859670 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62e04d93-8ab2-4879-8c4f-9c1210da5f20 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:08.919472 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a0ec2bd-516a-434c-95c0-7cb1bd51a4a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:08.920481 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a0ec2bd-516a-434c-95c0-7cb1bd51a4a0 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:08.921081 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a0ec2bd-516a-434c-95c0-7cb1bd51a4a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:08.921574 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a0ec2bd-516a-434c-95c0-7cb1bd51a4a0 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:08.942589 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a0ec2bd-516a-434c-95c0-7cb1bd51a4a0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:42:08.948345 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-7a0ec2bd-516a-434c-95c0-7cb1bd51a4a0 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 09 00:42:08.948345 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:42:08.948345 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:42:08.948345 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:42:08.948345 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:42:08.948345 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:42:08.948345 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:42:08.948345 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:42:08.948345 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:42:08.948345 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:42:08.948345 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:42:08.948345 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:42:08.948345 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:42:08.948345 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:42:08.948345 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:42:08.948345 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:42:08.949225 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:42:08.949225 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:42:08.949225 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:42:08.949225 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:42:08.949225 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:42:08.949225 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:42:08.949225 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:42:08.949225 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:42:08.949225 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:42:08.949225 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:42:08.949225 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:42:08.949225 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:42:08.949225 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:42:08.949225 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:42:08.949225 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:42:08.949225 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:42:08.949972 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:42:08.949972 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:42:08.949972 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:42:08.949972 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:42:08.949972 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:42:08.949972 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:42:08.949972 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:42:08.949972 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:42:08.949972 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:42:08.949972 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:42:08.949972 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 09 00:42:08.949972 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:42:08.953059 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 82/163] 10.222.0.33 () {56 vars in 1099 bytes} [Sun Oct 9 00:42:08 2022] GET /identity/v3/roles/member => generated 84 bytes in 246 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:42:08.958875 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a0ec2bd-516a-434c-95c0-7cb1bd51a4a0 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:09.055569 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aab2d6c-c16f-4109-b840-42d23c0f02e3 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:09.144247 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62a03740-8fae-4b1c-a6cf-62f308e9fd6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:09.145228 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a03740-8fae-4b1c-a6cf-62f308e9fd6a None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:09.145712 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a03740-8fae-4b1c-a6cf-62f308e9fd6a None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:09.146205 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a03740-8fae-4b1c-a6cf-62f308e9fd6a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:09.147133 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62a03740-8fae-4b1c-a6cf-62f308e9fd6a None admin] RBAC: Adding query filter params (role.id=092036ee46ac427baf4d56468575f697, scope.project.id=768d646302f443f1a7d490b8c3d4bdbb, user.id=d9c8218a78034540ba9ed09b697588a8) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:09.147748 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62a03740-8fae-4b1c-a6cf-62f308e9fd6a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:09.151671 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62a03740-8fae-4b1c-a6cf-62f308e9fd6a None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:09.163678 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 82/164] 10.222.0.33 () {56 vars in 1374 bytes} [Sun Oct 9 00:42:08 2022] GET /identity/v3/role_assignments?user.id=d9c8218a78034540ba9ed09b697588a8&scope.project.id=768d646302f443f1a7d490b8c3d4bdbb&role.id=092036ee46ac427baf4d56468575f697 => generated 608 bytes in 331 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:09.168698 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62a03740-8fae-4b1c-a6cf-62f308e9fd6a None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:09.196574 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a0ec2bd-516a-434c-95c0-7cb1bd51a4a0 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:09.243240 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:09.261681 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a20ab016-8922-4d2b-85a2-9c1cf79af1a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:09.262974 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a20ab016-8922-4d2b-85a2-9c1cf79af1a6 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:09.263528 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a20ab016-8922-4d2b-85a2-9c1cf79af1a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:09.264040 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a20ab016-8922-4d2b-85a2-9c1cf79af1a6 None admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:09.264978 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a20ab016-8922-4d2b-85a2-9c1cf79af1a6 None admin] RBAC: Adding query filter params (name=member) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:09.265482 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a20ab016-8922-4d2b-85a2-9c1cf79af1a6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:09.267104 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a20ab016-8922-4d2b-85a2-9c1cf79af1a6 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:09.277653 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 83/165] 10.222.0.33 () {56 vars in 1101 bytes} [Sun Oct 9 00:42:08 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 319 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:09.278999 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a20ab016-8922-4d2b-85a2-9c1cf79af1a6 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:09.318883 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:09.426500 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a20ab016-8922-4d2b-85a2-9c1cf79af1a6 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:09.461763 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62a03740-8fae-4b1c-a6cf-62f308e9fd6a None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:09.469501 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00feb57e-d02a-4d20-9419-a9268604ca7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:09.470305 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00feb57e-d02a-4d20-9419-a9268604ca7a None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:09.470689 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00feb57e-d02a-4d20-9419-a9268604ca7a None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:09.471064 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00feb57e-d02a-4d20-9419-a9268604ca7a None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:09.494051 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00feb57e-d02a-4d20-9419-a9268604ca7a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:42:09.499797 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-00feb57e-d02a-4d20-9419-a9268604ca7a None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 09 00:42:09.499797 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:42:09.499797 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:42:09.499797 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:42:09.499797 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:42:09.499797 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:42:09.499797 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:42:09.499797 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:42:09.499797 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:42:09.499797 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:42:09.499797 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:42:09.499797 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:42:09.499797 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:42:09.499797 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:42:09.499797 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:42:09.499797 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:42:09.500729 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:42:09.500729 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:42:09.500729 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:42:09.500729 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:42:09.500729 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:42:09.500729 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:42:09.500729 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:42:09.500729 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:42:09.500729 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:42:09.500729 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:42:09.500729 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:42:09.500729 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:42:09.500729 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:42:09.500729 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:42:09.500729 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:42:09.500729 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:42:09.501478 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:42:09.501478 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:42:09.501478 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:42:09.501478 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:42:09.501478 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:42:09.501478 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:42:09.501478 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:42:09.501478 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:42:09.501478 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:42:09.501478 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:42:09.501478 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 09 00:42:09.501478 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:42:09.507933 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 84/166] 10.222.0.33 () {56 vars in 1114 bytes} [Sun Oct 9 00:42:09 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 229 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:42:09.508851 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00feb57e-d02a-4d20-9419-a9268604ca7a None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:09.561240 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12794c4a-bb7c-4a04-a0e3-b68bf230810e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:09.561927 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12794c4a-bb7c-4a04-a0e3-b68bf230810e None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:09.562133 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12794c4a-bb7c-4a04-a0e3-b68bf230810e None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:09.562407 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12794c4a-bb7c-4a04-a0e3-b68bf230810e None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:09.563081 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12794c4a-bb7c-4a04-a0e3-b68bf230810e None admin] RBAC: Adding query filter params (name=member) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:09.563375 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12794c4a-bb7c-4a04-a0e3-b68bf230810e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:09.564801 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12794c4a-bb7c-4a04-a0e3-b68bf230810e None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:09.583787 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 83/167] 10.222.0.33 () {56 vars in 1101 bytes} [Sun Oct 9 00:42:09 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 411 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:09.584570 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12794c4a-bb7c-4a04-a0e3-b68bf230810e None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:09.591518 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:09.653658 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00feb57e-d02a-4d20-9419-a9268604ca7a None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:09.694104 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b286a01-bd8f-4e41-8bc9-23aa89d13468 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:09.695188 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b286a01-bd8f-4e41-8bc9-23aa89d13468 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:09.695729 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b286a01-bd8f-4e41-8bc9-23aa89d13468 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:09.696213 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b286a01-bd8f-4e41-8bc9-23aa89d13468 None admin] PATH_INFO: `/v3/users/da5fb3b5405648df8a2114ff2dca2d02` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:09.697563 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b286a01-bd8f-4e41-8bc9-23aa89d13468 None admin] RBAC: Authorizing `identity:get_user(user_id=da5fb3b5405648df8a2114ff2dca2d02)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:09.700351 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b286a01-bd8f-4e41-8bc9-23aa89d13468 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:09.702552 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 85/168] 10.222.0.33 () {56 vars in 1177 bytes} [Sun Oct 9 00:42:09 2022] GET /identity/v3/users/da5fb3b5405648df8a2114ff2dca2d02 => generated 275 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:09.703769 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b286a01-bd8f-4e41-8bc9-23aa89d13468 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:09.813664 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12794c4a-bb7c-4a04-a0e3-b68bf230810e None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:09.825088 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b286a01-bd8f-4e41-8bc9-23aa89d13468 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:09.839965 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:09.845283 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:09.873128 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a1a5202-5448-4fcd-864b-3ef189f92376 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:09.874205 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a1a5202-5448-4fcd-864b-3ef189f92376 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:09.874683 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a1a5202-5448-4fcd-864b-3ef189f92376 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:09.875187 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a1a5202-5448-4fcd-864b-3ef189f92376 None admin] PATH_INFO: `/v3/users/da5fb3b5405648df8a2114ff2dca2d02` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:09.877004 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a1a5202-5448-4fcd-864b-3ef189f92376 None admin] RBAC: Authorizing `identity:get_user(user_id=da5fb3b5405648df8a2114ff2dca2d02)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:09.879110 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a1a5202-5448-4fcd-864b-3ef189f92376 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:09.881003 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 86/169] 10.222.0.33 () {56 vars in 1177 bytes} [Sun Oct 9 00:42:09 2022] GET /identity/v3/users/da5fb3b5405648df8a2114ff2dca2d02 => generated 275 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:09.882101 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a1a5202-5448-4fcd-864b-3ef189f92376 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:09.887055 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e13de30d-a5c5-439a-b828-7e6c8980520a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:09.887705 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e13de30d-a5c5-439a-b828-7e6c8980520a None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:09.887993 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e13de30d-a5c5-439a-b828-7e6c8980520a None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:09.888269 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e13de30d-a5c5-439a-b828-7e6c8980520a None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:09.888940 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e13de30d-a5c5-439a-b828-7e6c8980520a None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:09.889245 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e13de30d-a5c5-439a-b828-7e6c8980520a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:09.890633 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e13de30d-a5c5-439a-b828-7e6c8980520a None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:09.906848 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 84/170] 10.222.0.33 () {56 vars in 1111 bytes} [Sun Oct 9 00:42:09 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 322 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:09.907505 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e13de30d-a5c5-439a-b828-7e6c8980520a None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:10.046201 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a1a5202-5448-4fcd-864b-3ef189f92376 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:10.067197 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e13de30d-a5c5-439a-b828-7e6c8980520a None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:10.101695 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98476162-75e1-4eb6-b324-d80970f2d6c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:10.102262 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98476162-75e1-4eb6-b324-d80970f2d6c5 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:10.102542 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98476162-75e1-4eb6-b324-d80970f2d6c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:10.102816 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98476162-75e1-4eb6-b324-d80970f2d6c5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:10.117847 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74c997ad-debe-4024-ab7f-4b77db9f903f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:10.118741 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c997ad-debe-4024-ab7f-4b77db9f903f None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:10.119317 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c997ad-debe-4024-ab7f-4b77db9f903f None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:10.120342 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c997ad-debe-4024-ab7f-4b77db9f903f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:10.157651 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:10.175738 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:10.177321 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98476162-75e1-4eb6-b324-d80970f2d6c5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:10.178798 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98476162-75e1-4eb6-b324-d80970f2d6c5 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:10.201914 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74c997ad-debe-4024-ab7f-4b77db9f903f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:10.203979 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74c997ad-debe-4024-ab7f-4b77db9f903f None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:10.285040 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 87/171] 10.222.0.33 () {58 vars in 1282 bytes} [Sun Oct 9 00:42:09 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 403 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 09 00:42:10.286020 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98476162-75e1-4eb6-b324-d80970f2d6c5 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:10.323700 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 85/172] 10.222.0.33 () {58 vars in 1282 bytes} [Sun Oct 9 00:42:09 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 416 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 09 00:42:10.325375 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9baf6ce0-075c-4777-833e-5f80ef5e2017 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:10.325867 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9baf6ce0-075c-4777-833e-5f80ef5e2017 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:10.326334 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9baf6ce0-075c-4777-833e-5f80ef5e2017 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:10.327869 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 86/173] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:42:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:42:10.328966 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9baf6ce0-075c-4777-833e-5f80ef5e2017 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:10.377637 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:10.404806 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98476162-75e1-4eb6-b324-d80970f2d6c5 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:10.418726 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:10.452489 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9baf6ce0-075c-4777-833e-5f80ef5e2017 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:10.461637 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceced4b1-1d43-413e-aefc-ba0c63308870 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:10.462553 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceced4b1-1d43-413e-aefc-ba0c63308870 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:10.464051 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceced4b1-1d43-413e-aefc-ba0c63308870 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:10.464529 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceced4b1-1d43-413e-aefc-ba0c63308870 None admin] PATH_INFO: `/v3/users/da5fb3b5405648df8a2114ff2dca2d02` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:10.466592 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceced4b1-1d43-413e-aefc-ba0c63308870 None admin] RBAC: Authorizing `identity:get_user(user_id=da5fb3b5405648df8a2114ff2dca2d02)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:10.468998 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceced4b1-1d43-413e-aefc-ba0c63308870 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:10.471365 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 88/174] 10.222.0.33 () {56 vars in 1177 bytes} [Sun Oct 9 00:42:10 2022] GET /identity/v3/users/da5fb3b5405648df8a2114ff2dca2d02 => generated 275 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:10.473237 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac4b2d26-02dc-4951-8a3f-aeb962aa8f75 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:10.473721 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac4b2d26-02dc-4951-8a3f-aeb962aa8f75 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:10.474202 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac4b2d26-02dc-4951-8a3f-aeb962aa8f75 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:10.489992 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dbc8f29-3e22-4f1b-b9e1-bc13691912dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:10.490619 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dbc8f29-3e22-4f1b-b9e1-bc13691912dd None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:10.490893 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dbc8f29-3e22-4f1b-b9e1-bc13691912dd None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:10.491174 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dbc8f29-3e22-4f1b-b9e1-bc13691912dd None admin] PATH_INFO: `/v3/projects/768d646302f443f1a7d490b8c3d4bdbb` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:10.492406 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dbc8f29-3e22-4f1b-b9e1-bc13691912dd None admin] RBAC: Authorizing `identity:get_project(project_id=768d646302f443f1a7d490b8c3d4bdbb)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:10.493839 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dbc8f29-3e22-4f1b-b9e1-bc13691912dd None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:10.495488 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 87/175] 10.222.0.33 () {56 vars in 1186 bytes} [Sun Oct 9 00:42:10 2022] GET /identity/v3/projects/768d646302f443f1a7d490b8c3d4bdbb => generated 298 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:10.496280 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dbc8f29-3e22-4f1b-b9e1-bc13691912dd None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:10.498148 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-ac4b2d26-02dc-4951-8a3f-aeb962aa8f75 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:42:10.500896 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-ac4b2d26-02dc-4951-8a3f-aeb962aa8f75 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:42:10.592462 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 89/176] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:42:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 09 00:42:10.594565 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d1919b-3e9a-4f5f-9f5e-e655aa0136d7 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:10.594991 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d1919b-3e9a-4f5f-9f5e-e655aa0136d7 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:10.595807 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d1919b-3e9a-4f5f-9f5e-e655aa0136d7 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:10.603488 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 90/177] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:42:10 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:42:10.603488 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4d1919b-3e9a-4f5f-9f5e-e655aa0136d7 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:10.626025 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dbc8f29-3e22-4f1b-b9e1-bc13691912dd None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:10.681788 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d09c14fa-141a-4f5d-9ad9-f83f425863c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:10.683810 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d09c14fa-141a-4f5d-9ad9-f83f425863c4 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:10.684082 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d09c14fa-141a-4f5d-9ad9-f83f425863c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:10.684351 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d09c14fa-141a-4f5d-9ad9-f83f425863c4 None admin] PATH_INFO: `/v3/projects/fc09cf0f89194e89b47f7e4e63ad25da` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:10.686376 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d09c14fa-141a-4f5d-9ad9-f83f425863c4 None admin] RBAC: Authorizing `identity:get_project(project_id=fc09cf0f89194e89b47f7e4e63ad25da)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:10.688026 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d09c14fa-141a-4f5d-9ad9-f83f425863c4 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:10.692462 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 88/178] 10.222.0.33 () {56 vars in 1186 bytes} [Sun Oct 9 00:42:10 2022] GET /identity/v3/projects/fc09cf0f89194e89b47f7e4e63ad25da => generated 312 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:10.696173 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d09c14fa-141a-4f5d-9ad9-f83f425863c4 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:10.704775 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:10.726676 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:10.740768 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4d1919b-3e9a-4f5f-9f5e-e655aa0136d7 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:10.812339 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97be995e-8901-4622-b873-dc97592ce0bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:10.814205 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97be995e-8901-4622-b873-dc97592ce0bd None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:10.814483 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97be995e-8901-4622-b873-dc97592ce0bd None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:10.814748 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97be995e-8901-4622-b873-dc97592ce0bd None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:10.858872 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d09c14fa-141a-4f5d-9ad9-f83f425863c4 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:10.901688 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97be995e-8901-4622-b873-dc97592ce0bd None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:10.904173 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97be995e-8901-4622-b873-dc97592ce0bd None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:10.916137 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e59e7976-a14d-49fd-8a49-202ad0b50aaf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:10.917311 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e59e7976-a14d-49fd-8a49-202ad0b50aaf None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:10.917590 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e59e7976-a14d-49fd-8a49-202ad0b50aaf None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:10.917862 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e59e7976-a14d-49fd-8a49-202ad0b50aaf None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:10.918592 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e59e7976-a14d-49fd-8a49-202ad0b50aaf None admin] RBAC: Adding query filter params (role.id=588016933bab436d8b7c3ac76827356d, scope.project.id=768d646302f443f1a7d490b8c3d4bdbb, user.id=da5fb3b5405648df8a2114ff2dca2d02) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:10.918891 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e59e7976-a14d-49fd-8a49-202ad0b50aaf None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:10.920367 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e59e7976-a14d-49fd-8a49-202ad0b50aaf None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:10.927359 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 89/179] 10.222.0.33 () {56 vars in 1374 bytes} [Sun Oct 9 00:42:10 2022] GET /identity/v3/role_assignments?user.id=da5fb3b5405648df8a2114ff2dca2d02&scope.project.id=768d646302f443f1a7d490b8c3d4bdbb&role.id=588016933bab436d8b7c3ac76827356d => generated 608 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:10.928358 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc947e9-1edf-461d-896c-b351cf8d4615 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:10.928644 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc947e9-1edf-461d-896c-b351cf8d4615 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:10.928918 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc947e9-1edf-461d-896c-b351cf8d4615 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:10.974170 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-5dc947e9-1edf-461d-896c-b351cf8d4615 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:42:10.976678 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-5dc947e9-1edf-461d-896c-b351cf8d4615 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:42:11.043855 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:11.066880 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 91/180] 10.222.0.33 () {58 vars in 1282 bytes} [Sun Oct 9 00:42:10 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 466 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 09 00:42:11.068172 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97be995e-8901-4622-b873-dc97592ce0bd None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:11.082025 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 90/181] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:42:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 09 00:42:11.083124 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dc947e9-1edf-461d-896c-b351cf8d4615 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:11.108044 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:11.199801 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97be995e-8901-4622-b873-dc97592ce0bd None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:11.226298 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dc947e9-1edf-461d-896c-b351cf8d4615 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:11.245891 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5087347-63bc-4253-a9de-6bd68709583c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:11.246673 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5087347-63bc-4253-a9de-6bd68709583c None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:11.246959 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5087347-63bc-4253-a9de-6bd68709583c None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:11.247237 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5087347-63bc-4253-a9de-6bd68709583c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:11.248543 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5087347-63bc-4253-a9de-6bd68709583c None admin] RBAC: Adding query filter params (role.id=588016933bab436d8b7c3ac76827356d, scope.project.id=fc09cf0f89194e89b47f7e4e63ad25da, user.id=da5fb3b5405648df8a2114ff2dca2d02) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:11.248973 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5087347-63bc-4253-a9de-6bd68709583c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:11.250637 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5087347-63bc-4253-a9de-6bd68709583c None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:11.258710 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 92/182] 10.222.0.33 () {56 vars in 1374 bytes} [Sun Oct 9 00:42:11 2022] GET /identity/v3/role_assignments?user.id=da5fb3b5405648df8a2114ff2dca2d02&scope.project.id=fc09cf0f89194e89b47f7e4e63ad25da&role.id=588016933bab436d8b7c3ac76827356d => generated 608 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:11.259950 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae255ad3-1c78-4565-88c7-ef54b745bef0 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:11.260958 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae255ad3-1c78-4565-88c7-ef54b745bef0 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:11.261237 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae255ad3-1c78-4565-88c7-ef54b745bef0 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:11.263306 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 93/183] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:42:11 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:42:11.264097 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae255ad3-1c78-4565-88c7-ef54b745bef0 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:11.270450 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e6e790a-d8ca-42d6-a289-b874f68b6e8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:11.271247 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e6e790a-d8ca-42d6-a289-b874f68b6e8c None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:11.271533 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e6e790a-d8ca-42d6-a289-b874f68b6e8c None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:11.271843 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e6e790a-d8ca-42d6-a289-b874f68b6e8c None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:11.284081 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e6e790a-d8ca-42d6-a289-b874f68b6e8c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:42:11.289620 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-3e6e790a-d8ca-42d6-a289-b874f68b6e8c None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 09 00:42:11.289620 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:42:11.289620 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:42:11.289620 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:42:11.289620 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:42:11.289620 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:42:11.289620 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:42:11.289620 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:42:11.289620 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:42:11.289620 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:42:11.289620 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:42:11.289620 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:42:11.289620 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:42:11.289620 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:42:11.289620 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:42:11.289620 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:42:11.290538 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:42:11.290538 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:42:11.290538 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:42:11.290538 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:42:11.290538 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:42:11.290538 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:42:11.290538 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:42:11.290538 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:42:11.290538 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:42:11.290538 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:42:11.290538 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:42:11.290538 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:42:11.290538 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:42:11.290538 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:42:11.290538 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:42:11.290538 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:42:11.291795 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:42:11.291795 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:42:11.291795 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:42:11.291795 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:42:11.291795 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:42:11.291795 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:42:11.291795 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:42:11.291795 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:42:11.291795 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:42:11.291795 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:42:11.291795 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 09 00:42:11.291795 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:42:11.291795 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 91/184] 10.222.0.33 () {56 vars in 1099 bytes} [Sun Oct 9 00:42:11 2022] GET /identity/v3/roles/reader => generated 84 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:42:11.292453 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b216207-2a36-486a-baa1-e62fe58b71d7 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:11.292453 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b216207-2a36-486a-baa1-e62fe58b71d7 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:11.292818 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b216207-2a36-486a-baa1-e62fe58b71d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:11.319897 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-4b216207-2a36-486a-baa1-e62fe58b71d7 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:42:11.321126 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-4b216207-2a36-486a-baa1-e62fe58b71d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:42:11.331210 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:11.405000 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae255ad3-1c78-4565-88c7-ef54b745bef0 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:11.447969 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:11.450099 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87d93741-d845-4248-ab59-345d1a244b9a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:11.450254 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d93741-d845-4248-ab59-345d1a244b9a None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:11.450520 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d93741-d845-4248-ab59-345d1a244b9a None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:11.450812 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d93741-d845-4248-ab59-345d1a244b9a None admin] PATH_INFO: `/v3/projects/768d646302f443f1a7d490b8c3d4bdbb` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:11.452152 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87d93741-d845-4248-ab59-345d1a244b9a None admin] RBAC: Authorizing `identity:get_project(project_id=768d646302f443f1a7d490b8c3d4bdbb)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:11.453968 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87d93741-d845-4248-ab59-345d1a244b9a None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:11.455888 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 94/185] 10.222.0.33 () {56 vars in 1186 bytes} [Sun Oct 9 00:42:11 2022] GET /identity/v3/projects/768d646302f443f1a7d490b8c3d4bdbb => generated 298 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:11.456694 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87d93741-d845-4248-ab59-345d1a244b9a None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:11.457373 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 92/186] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:42:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 09 00:42:11.459508 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b216207-2a36-486a-baa1-e62fe58b71d7 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:11.588245 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b216207-2a36-486a-baa1-e62fe58b71d7 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:11.596093 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87d93741-d845-4248-ab59-345d1a244b9a None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:11.609479 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:11.629192 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-313857b1-b569-4075-9eb0-4a0cc1c74458 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:11.629926 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313857b1-b569-4075-9eb0-4a0cc1c74458 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:11.630235 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313857b1-b569-4075-9eb0-4a0cc1c74458 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:11.630483 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313857b1-b569-4075-9eb0-4a0cc1c74458 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:11.632973 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c40707d9-c8c8-48d2-8d9f-d0d678dfcaef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:11.633853 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c40707d9-c8c8-48d2-8d9f-d0d678dfcaef None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:11.634398 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c40707d9-c8c8-48d2-8d9f-d0d678dfcaef None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:11.634680 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c40707d9-c8c8-48d2-8d9f-d0d678dfcaef None admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:11.635995 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c40707d9-c8c8-48d2-8d9f-d0d678dfcaef None admin] RBAC: Adding query filter params (name=reader) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:11.636300 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c40707d9-c8c8-48d2-8d9f-d0d678dfcaef None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:11.637674 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c40707d9-c8c8-48d2-8d9f-d0d678dfcaef None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:11.642894 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-313857b1-b569-4075-9eb0-4a0cc1c74458 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:42:11.646560 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 95/187] 10.222.0.33 () {56 vars in 1101 bytes} [Sun Oct 9 00:42:11 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:11.647389 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c40707d9-c8c8-48d2-8d9f-d0d678dfcaef None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:11.648565 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-313857b1-b569-4075-9eb0-4a0cc1c74458 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 09 00:42:11.648565 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:42:11.648565 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:42:11.648565 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:42:11.648565 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:42:11.648565 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:42:11.648565 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:42:11.648565 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:42:11.648565 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:42:11.648565 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:42:11.648565 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:42:11.648565 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:42:11.648565 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:42:11.648565 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:42:11.648565 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:42:11.648565 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:42:11.649374 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:42:11.649374 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:42:11.649374 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:42:11.649374 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:42:11.649374 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:42:11.649374 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:42:11.649374 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:42:11.649374 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:42:11.649374 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:42:11.649374 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:42:11.649374 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:42:11.649374 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:42:11.649374 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:42:11.649374 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:42:11.649374 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:42:11.649374 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:42:11.650515 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:42:11.650515 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:42:11.650515 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:42:11.650515 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:42:11.650515 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:42:11.650515 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:42:11.650515 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:42:11.650515 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:42:11.650515 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:42:11.650515 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:42:11.650515 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 09 00:42:11.650515 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:42:11.650515 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 93/188] 10.222.0.33 () {56 vars in 1096 bytes} [Sun Oct 9 00:42:11 2022] GET /identity/v3/roles/admin => generated 83 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:42:11.655612 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-313857b1-b569-4075-9eb0-4a0cc1c74458 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:11.721820 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:11.761845 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c40707d9-c8c8-48d2-8d9f-d0d678dfcaef None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:11.766809 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-313857b1-b569-4075-9eb0-4a0cc1c74458 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:11.796294 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f10fe21-3d4a-4d05-a393-0891f0f7bcc8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:11.796895 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f10fe21-3d4a-4d05-a393-0891f0f7bcc8 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:11.797170 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f10fe21-3d4a-4d05-a393-0891f0f7bcc8 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:11.797441 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f10fe21-3d4a-4d05-a393-0891f0f7bcc8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:11.798152 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f10fe21-3d4a-4d05-a393-0891f0f7bcc8 None admin] RBAC: Adding query filter params (role.id=7c1fdac272a34be99661077b0f1b8014, scope.project.id=768d646302f443f1a7d490b8c3d4bdbb, user.id=da5fb3b5405648df8a2114ff2dca2d02) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:11.798454 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f10fe21-3d4a-4d05-a393-0891f0f7bcc8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:11.799918 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f10fe21-3d4a-4d05-a393-0891f0f7bcc8 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:11.802414 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-481dbc3b-5205-4a73-8748-199f87d27875 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:11.803015 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-481dbc3b-5205-4a73-8748-199f87d27875 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:11.803294 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-481dbc3b-5205-4a73-8748-199f87d27875 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:11.803569 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-481dbc3b-5205-4a73-8748-199f87d27875 None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:11.804256 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-481dbc3b-5205-4a73-8748-199f87d27875 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:11.804556 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-481dbc3b-5205-4a73-8748-199f87d27875 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:11.805531 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 96/189] 10.222.0.33 () {56 vars in 1374 bytes} [Sun Oct 9 00:42:11 2022] GET /identity/v3/role_assignments?user.id=da5fb3b5405648df8a2114ff2dca2d02&scope.project.id=768d646302f443f1a7d490b8c3d4bdbb&role.id=7c1fdac272a34be99661077b0f1b8014 => generated 608 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:11.805936 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-481dbc3b-5205-4a73-8748-199f87d27875 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:11.806289 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f10fe21-3d4a-4d05-a393-0891f0f7bcc8 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:11.824156 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 94/190] 10.222.0.33 () {56 vars in 1099 bytes} [Sun Oct 9 00:42:11 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:11.825482 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056db4dc-92df-4844-ba53-7d8687dfa6cc None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:11.825766 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056db4dc-92df-4844-ba53-7d8687dfa6cc None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:11.826037 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056db4dc-92df-4844-ba53-7d8687dfa6cc None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:11.827800 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 95/191] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:42:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:42:11.828167 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-056db4dc-92df-4844-ba53-7d8687dfa6cc None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:11.860385 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:11.928453 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f10fe21-3d4a-4d05-a393-0891f0f7bcc8 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:11.952187 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-056db4dc-92df-4844-ba53-7d8687dfa6cc None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:11.967944 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c825ed91-17eb-485a-bb77-b7a58cb2a74f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:11.968620 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c825ed91-17eb-485a-bb77-b7a58cb2a74f None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:11.968899 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c825ed91-17eb-485a-bb77-b7a58cb2a74f None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:11.969170 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c825ed91-17eb-485a-bb77-b7a58cb2a74f None admin] PATH_INFO: `/v3/users/d90f753f3b2e4bb08eb926d2d604138e` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:11.970423 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c825ed91-17eb-485a-bb77-b7a58cb2a74f None admin] RBAC: Authorizing `identity:get_user(user_id=d90f753f3b2e4bb08eb926d2d604138e)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:11.972064 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c825ed91-17eb-485a-bb77-b7a58cb2a74f None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:11.973796 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:11.974469 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 97/192] 10.222.0.33 () {56 vars in 1177 bytes} [Sun Oct 9 00:42:11 2022] GET /identity/v3/users/d90f753f3b2e4bb08eb926d2d604138e => generated 289 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:11.975084 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c825ed91-17eb-485a-bb77-b7a58cb2a74f None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:11.993658 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b9bb26a-e19b-4825-a69f-15f44284f52c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:11.994107 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b9bb26a-e19b-4825-a69f-15f44284f52c None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:11.994409 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b9bb26a-e19b-4825-a69f-15f44284f52c None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:11.995233 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b9bb26a-e19b-4825-a69f-15f44284f52c None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:12.011681 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b9bb26a-e19b-4825-a69f-15f44284f52c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:42:12.016509 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-3b9bb26a-e19b-4825-a69f-15f44284f52c None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 09 00:42:12.016509 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:42:12.016509 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:42:12.016509 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:42:12.016509 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:42:12.016509 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:42:12.016509 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:42:12.016509 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:42:12.016509 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:42:12.016509 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:42:12.016509 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:42:12.016509 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:42:12.016509 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:42:12.016509 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:42:12.016509 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:42:12.016509 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:42:12.017346 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:42:12.017346 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:42:12.017346 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:42:12.017346 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:42:12.017346 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:42:12.017346 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:42:12.017346 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:42:12.017346 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:42:12.017346 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:42:12.017346 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:42:12.017346 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:42:12.017346 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:42:12.017346 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:42:12.017346 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:42:12.017346 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:42:12.017346 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:42:12.018390 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:42:12.018390 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:42:12.018390 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:42:12.018390 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:42:12.018390 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:42:12.018390 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:42:12.018390 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:42:12.018390 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:42:12.018390 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:42:12.018390 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:42:12.018390 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 09 00:42:12.018390 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:42:12.018390 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 96/193] 10.222.0.33 () {56 vars in 1114 bytes} [Sun Oct 9 00:42:11 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:42:12.019043 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676a6af4-edff-4fda-8696-b817cdf937b3 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:12.019043 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676a6af4-edff-4fda-8696-b817cdf937b3 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:12.019274 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676a6af4-edff-4fda-8696-b817cdf937b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:12.038194 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-676a6af4-edff-4fda-8696-b817cdf937b3 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:42:12.039646 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-676a6af4-edff-4fda-8696-b817cdf937b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:42:12.094773 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c825ed91-17eb-485a-bb77-b7a58cb2a74f None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:12.102085 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:12.115825 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 97/194] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:42:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 09 00:42:12.116872 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-676a6af4-edff-4fda-8696-b817cdf937b3 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:12.134644 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14a8f536-d493-4a43-bfff-aefd34a71e01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:12.135353 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14a8f536-d493-4a43-bfff-aefd34a71e01 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:12.135681 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14a8f536-d493-4a43-bfff-aefd34a71e01 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:12.135961 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14a8f536-d493-4a43-bfff-aefd34a71e01 None admin] PATH_INFO: `/v3/users/6c8f41499fe54de28740b35e4f20cf88` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:12.149686 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14a8f536-d493-4a43-bfff-aefd34a71e01 None admin] RBAC: Authorizing `identity:get_user(user_id=6c8f41499fe54de28740b35e4f20cf88)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:12.151716 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14a8f536-d493-4a43-bfff-aefd34a71e01 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:12.153341 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 98/195] 10.222.0.33 () {56 vars in 1177 bytes} [Sun Oct 9 00:42:11 2022] GET /identity/v3/users/6c8f41499fe54de28740b35e4f20cf88 => generated 283 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:12.154094 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14a8f536-d493-4a43-bfff-aefd34a71e01 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:12.239319 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-676a6af4-edff-4fda-8696-b817cdf937b3 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:12.253199 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:12.260303 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14a8f536-d493-4a43-bfff-aefd34a71e01 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:12.278978 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55806991-88e7-43d7-8273-c448136467fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:12.279579 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55806991-88e7-43d7-8273-c448136467fb None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:12.279884 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55806991-88e7-43d7-8273-c448136467fb None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:12.280153 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55806991-88e7-43d7-8273-c448136467fb None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:12.299100 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a8c34d9-2f8f-4c16-9408-b42bce59155c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:12.299919 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a8c34d9-2f8f-4c16-9408-b42bce59155c None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:12.300026 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a8c34d9-2f8f-4c16-9408-b42bce59155c None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:12.300262 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a8c34d9-2f8f-4c16-9408-b42bce59155c None admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:12.300913 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a8c34d9-2f8f-4c16-9408-b42bce59155c None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:12.301214 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a8c34d9-2f8f-4c16-9408-b42bce59155c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:12.302576 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a8c34d9-2f8f-4c16-9408-b42bce59155c None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:12.315109 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 99/196] 10.222.0.33 () {56 vars in 1111 bytes} [Sun Oct 9 00:42:12 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:12.315109 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a8c34d9-2f8f-4c16-9408-b42bce59155c None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:12.343024 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:12.358292 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55806991-88e7-43d7-8273-c448136467fb None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:12.359795 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55806991-88e7-43d7-8273-c448136467fb None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:12.419023 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a8c34d9-2f8f-4c16-9408-b42bce59155c None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:12.451134 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76d05e65-4cf8-4fd9-8138-33fcfcdc098d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:12.451766 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76d05e65-4cf8-4fd9-8138-33fcfcdc098d None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:12.452043 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76d05e65-4cf8-4fd9-8138-33fcfcdc098d None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:12.452313 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76d05e65-4cf8-4fd9-8138-33fcfcdc098d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:12.458672 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 98/197] 10.222.0.33 () {58 vars in 1282 bytes} [Sun Oct 9 00:42:12 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 342 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 09 00:42:12.459496 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55806991-88e7-43d7-8273-c448136467fb None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:12.473062 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:12.517308 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76d05e65-4cf8-4fd9-8138-33fcfcdc098d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:12.518679 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76d05e65-4cf8-4fd9-8138-33fcfcdc098d None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:12.544388 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:12.560220 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55806991-88e7-43d7-8273-c448136467fb None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:12.593522 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b3adfcd-90b3-4763-863b-b0941c64198f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:12.594147 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b3adfcd-90b3-4763-863b-b0941c64198f None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:12.594426 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b3adfcd-90b3-4763-863b-b0941c64198f None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:12.594698 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b3adfcd-90b3-4763-863b-b0941c64198f None admin] PATH_INFO: `/v3/users/6c8f41499fe54de28740b35e4f20cf88` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:12.595844 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b3adfcd-90b3-4763-863b-b0941c64198f None admin] RBAC: Authorizing `identity:get_user(user_id=6c8f41499fe54de28740b35e4f20cf88)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:12.597284 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b3adfcd-90b3-4763-863b-b0941c64198f None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:12.599005 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 99/198] 10.222.0.33 () {56 vars in 1177 bytes} [Sun Oct 9 00:42:12 2022] GET /identity/v3/users/6c8f41499fe54de28740b35e4f20cf88 => generated 283 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:12.599608 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b3adfcd-90b3-4763-863b-b0941c64198f None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:12.619510 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 100/199] 10.222.0.33 () {58 vars in 1282 bytes} [Sun Oct 9 00:42:12 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 09 00:42:12.621013 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76d05e65-4cf8-4fd9-8138-33fcfcdc098d None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:12.675648 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:12.698127 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b3adfcd-90b3-4763-863b-b0941c64198f None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:12.718659 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76d05e65-4cf8-4fd9-8138-33fcfcdc098d None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:12.733100 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f30ddc70-e365-4d7a-8b07-6512d6084fdf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:12.733723 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f30ddc70-e365-4d7a-8b07-6512d6084fdf None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:12.734002 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f30ddc70-e365-4d7a-8b07-6512d6084fdf None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:12.734274 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f30ddc70-e365-4d7a-8b07-6512d6084fdf None admin] PATH_INFO: `/v3/domains/default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:12.735377 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f30ddc70-e365-4d7a-8b07-6512d6084fdf None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:12.736873 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f30ddc70-e365-4d7a-8b07-6512d6084fdf None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:12.738478 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 100/200] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:42:12 2022] GET /identity/v3/domains/default => generated 199 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:12.739397 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f30ddc70-e365-4d7a-8b07-6512d6084fdf None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:12.747806 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:12.756255 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d14d916b-0c98-4ab3-8d57-39ae82246937 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:12.756883 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d14d916b-0c98-4ab3-8d57-39ae82246937 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:12.757168 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d14d916b-0c98-4ab3-8d57-39ae82246937 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:12.757439 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d14d916b-0c98-4ab3-8d57-39ae82246937 None admin] PATH_INFO: `/v3/projects/768d646302f443f1a7d490b8c3d4bdbb` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:12.758573 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d14d916b-0c98-4ab3-8d57-39ae82246937 None admin] RBAC: Authorizing `identity:get_project(project_id=768d646302f443f1a7d490b8c3d4bdbb)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:12.760567 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d14d916b-0c98-4ab3-8d57-39ae82246937 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:12.762313 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 101/201] 10.222.0.33 () {56 vars in 1186 bytes} [Sun Oct 9 00:42:12 2022] GET /identity/v3/projects/768d646302f443f1a7d490b8c3d4bdbb => generated 298 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:12.762920 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d14d916b-0c98-4ab3-8d57-39ae82246937 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:12.842756 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f30ddc70-e365-4d7a-8b07-6512d6084fdf None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:12.857797 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d14d916b-0c98-4ab3-8d57-39ae82246937 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:12.875998 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5774753c-a578-449e-a9c3-93a118f2fd3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:12.876589 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5774753c-a578-449e-a9c3-93a118f2fd3c None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:12.876865 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5774753c-a578-449e-a9c3-93a118f2fd3c None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:12.877136 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5774753c-a578-449e-a9c3-93a118f2fd3c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:12.891474 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-095ce61b-b89c-4d39-be42-0d8b0019b619 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:12.892120 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-095ce61b-b89c-4d39-be42-0d8b0019b619 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:12.892400 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-095ce61b-b89c-4d39-be42-0d8b0019b619 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:12.892676 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-095ce61b-b89c-4d39-be42-0d8b0019b619 None admin] PATH_INFO: `/v3/projects/dd8011750cb34075a6f8106533f848c2` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:12.893084 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:12.893907 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-095ce61b-b89c-4d39-be42-0d8b0019b619 None admin] RBAC: Authorizing `identity:get_project(project_id=dd8011750cb34075a6f8106533f848c2)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:12.895381 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-095ce61b-b89c-4d39-be42-0d8b0019b619 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:12.901872 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 102/202] 10.222.0.33 () {56 vars in 1186 bytes} [Sun Oct 9 00:42:12 2022] GET /identity/v3/projects/dd8011750cb34075a6f8106533f848c2 => generated 302 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:12.901872 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-095ce61b-b89c-4d39-be42-0d8b0019b619 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:12.947414 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5774753c-a578-449e-a9c3-93a118f2fd3c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:12.949482 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5774753c-a578-449e-a9c3-93a118f2fd3c None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:12.956660 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:12.993813 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-095ce61b-b89c-4d39-be42-0d8b0019b619 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:13.027455 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7a56611-7976-4ef1-bbb8-eb0fdc7e9ed3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:13.028074 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7a56611-7976-4ef1-bbb8-eb0fdc7e9ed3 None admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:13.028356 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7a56611-7976-4ef1-bbb8-eb0fdc7e9ed3 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:13.028628 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7a56611-7976-4ef1-bbb8-eb0fdc7e9ed3 None admin] PATH_INFO: `/v3/users` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:13.029268 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7a56611-7976-4ef1-bbb8-eb0fdc7e9ed3 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:13.030692 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7a56611-7976-4ef1-bbb8-eb0fdc7e9ed3 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:13.047601 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 101/203] 10.222.0.33 () {58 vars in 1282 bytes} [Sun Oct 9 00:42:12 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 09 00:42:13.048450 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5774753c-a578-449e-a9c3-93a118f2fd3c None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:13.051626 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 103/204] 10.222.0.33 () {60 vars in 1132 bytes} [Sun Oct 9 00:42:12 2022] POST /identity/v3/users => generated 297 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:42:13.051626 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7a56611-7976-4ef1-bbb8-eb0fdc7e9ed3 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:13.112502 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:13.173933 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7a56611-7976-4ef1-bbb8-eb0fdc7e9ed3 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:13.175392 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5774753c-a578-449e-a9c3-93a118f2fd3c None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:13.193365 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:13.212660 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33ba8a97-917b-451f-95b5-5b73133c3a97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:13.213497 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ba8a97-917b-451f-95b5-5b73133c3a97 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:13.213843 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ba8a97-917b-451f-95b5-5b73133c3a97 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:13.214181 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ba8a97-917b-451f-95b5-5b73133c3a97 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:13.214989 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33ba8a97-917b-451f-95b5-5b73133c3a97 None admin] RBAC: Adding query filter params (role.id=092036ee46ac427baf4d56468575f697, scope.project.id=dd8011750cb34075a6f8106533f848c2, user.id=6c8f41499fe54de28740b35e4f20cf88) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:13.215336 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33ba8a97-917b-451f-95b5-5b73133c3a97 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:13.215628 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f8b24c3-7745-4f00-8cca-53d3115f0520 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:13.216237 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8b24c3-7745-4f00-8cca-53d3115f0520 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:13.216517 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8b24c3-7745-4f00-8cca-53d3115f0520 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:13.216789 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8b24c3-7745-4f00-8cca-53d3115f0520 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:13.217144 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33ba8a97-917b-451f-95b5-5b73133c3a97 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:13.217517 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f8b24c3-7745-4f00-8cca-53d3115f0520 None admin] RBAC: Adding query filter params (role.id=2d631b6eec1646c9834f74efa1d66091, scope.project.id=768d646302f443f1a7d490b8c3d4bdbb, user.id=d90f753f3b2e4bb08eb926d2d604138e) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:13.218491 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f8b24c3-7745-4f00-8cca-53d3115f0520 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:13.219956 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f8b24c3-7745-4f00-8cca-53d3115f0520 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:13.223187 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 104/205] 10.222.0.33 () {56 vars in 1374 bytes} [Sun Oct 9 00:42:13 2022] GET /identity/v3/role_assignments?user.id=6c8f41499fe54de28740b35e4f20cf88&scope.project.id=dd8011750cb34075a6f8106533f848c2&role.id=092036ee46ac427baf4d56468575f697 => generated 259 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:13.233364 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 102/206] 10.222.0.33 () {56 vars in 1374 bytes} [Sun Oct 9 00:42:13 2022] GET /identity/v3/role_assignments?user.id=d90f753f3b2e4bb08eb926d2d604138e&scope.project.id=768d646302f443f1a7d490b8c3d4bdbb&role.id=2d631b6eec1646c9834f74efa1d66091 => generated 608 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:13.233580 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33ba8a97-917b-451f-95b5-5b73133c3a97 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:13.371186 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33ba8a97-917b-451f-95b5-5b73133c3a97 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:13.422307 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae360f7d-637f-430a-926c-12c377233cc5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:13.422960 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae360f7d-637f-430a-926c-12c377233cc5 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:13.423246 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae360f7d-637f-430a-926c-12c377233cc5 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:13.423523 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae360f7d-637f-430a-926c-12c377233cc5 None admin] PATH_INFO: `/v3/projects/dd8011750cb34075a6f8106533f848c2` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:13.424745 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae360f7d-637f-430a-926c-12c377233cc5 None admin] RBAC: Authorizing `identity:get_project(project_id=dd8011750cb34075a6f8106533f848c2)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:13.426273 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae360f7d-637f-430a-926c-12c377233cc5 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:13.428027 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 105/207] 10.222.0.33 () {56 vars in 1186 bytes} [Sun Oct 9 00:42:13 2022] GET /identity/v3/projects/dd8011750cb34075a6f8106533f848c2 => generated 302 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:13.434550 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f8b24c3-7745-4f00-8cca-53d3115f0520 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:13.545691 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f8b24c3-7745-4f00-8cca-53d3115f0520 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:13.556894 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:13.582323 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-397d8cea-c98b-4db9-b1da-09210409eded None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:13.582995 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-397d8cea-c98b-4db9-b1da-09210409eded None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:13.583279 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-397d8cea-c98b-4db9-b1da-09210409eded None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:13.583554 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-397d8cea-c98b-4db9-b1da-09210409eded None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:13.584365 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-397d8cea-c98b-4db9-b1da-09210409eded None admin] RBAC: Adding query filter params (role.id=7c1fdac272a34be99661077b0f1b8014, scope.project.id=dd8011750cb34075a6f8106533f848c2, user.id=6c8f41499fe54de28740b35e4f20cf88) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:13.584681 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-397d8cea-c98b-4db9-b1da-09210409eded None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:13.586135 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-397d8cea-c98b-4db9-b1da-09210409eded None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:13.592831 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 103/208] 10.222.0.33 () {56 vars in 1374 bytes} [Sun Oct 9 00:42:13 2022] GET /identity/v3/role_assignments?user.id=6c8f41499fe54de28740b35e4f20cf88&scope.project.id=dd8011750cb34075a6f8106533f848c2&role.id=7c1fdac272a34be99661077b0f1b8014 => generated 259 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:15.462333 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5690558f-783e-4bd0-a45b-06cf303d750c None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:15.463450 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5690558f-783e-4bd0-a45b-06cf303d750c None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:15.464016 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5690558f-783e-4bd0-a45b-06cf303d750c None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:15.465703 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 106/209] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:42:15 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:42:15.471786 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df4dc89-d96c-4419-b11f-703dda9e83ed None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:15.472784 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df4dc89-d96c-4419-b11f-703dda9e83ed None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:15.473279 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df4dc89-d96c-4419-b11f-703dda9e83ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:15.495386 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d44266a8-7ef2-426d-996d-3a3b33f3cc35 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:15.495739 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-7df4dc89-d96c-4419-b11f-703dda9e83ed None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:42:15.496219 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d44266a8-7ef2-426d-996d-3a3b33f3cc35 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:15.496697 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d44266a8-7ef2-426d-996d-3a3b33f3cc35 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:15.496934 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-7df4dc89-d96c-4419-b11f-703dda9e83ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:42:15.498457 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 107/210] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:42:15 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:42:15.504144 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1820e2b-b748-41c2-a861-49281383a882 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:15.504664 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1820e2b-b748-41c2-a861-49281383a882 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:15.505138 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1820e2b-b748-41c2-a861-49281383a882 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:15.529595 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-e1820e2b-b748-41c2-a861-49281383a882 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:42:15.531790 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-e1820e2b-b748-41c2-a861-49281383a882 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:42:15.573810 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 104/211] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:42:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 09 00:42:15.580317 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:15.601386 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 108/212] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:42:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 09 00:42:15.602135 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1226d3-4f9d-4704-8a0b-49b1201b1b0b None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:15.602666 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1226d3-4f9d-4704-8a0b-49b1201b1b0b None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:15.603161 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1226d3-4f9d-4704-8a0b-49b1201b1b0b None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:15.605267 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 105/213] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:42:15 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 09 00:42:15.610622 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9691099f-3580-44de-94f6-e2835410e92d None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:15.610907 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9691099f-3580-44de-94f6-e2835410e92d None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:15.611183 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9691099f-3580-44de-94f6-e2835410e92d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:15.629653 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-9691099f-3580-44de-94f6-e2835410e92d None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:42:15.630849 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-9691099f-3580-44de-94f6-e2835410e92d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:42:15.711727 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 109/214] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:42:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 09 00:42:15.760615 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5503d7e9-d037-4f7e-804f-ba5cb5d9220c None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:15.760900 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5503d7e9-d037-4f7e-804f-ba5cb5d9220c None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:15.761098 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5503d7e9-d037-4f7e-804f-ba5cb5d9220c None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:15.762461 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 106/215] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:42:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:42:15.767789 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a980fd0-c764-477e-b0e8-655fccc406bb None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:15.768249 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a980fd0-c764-477e-b0e8-655fccc406bb None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:15.768673 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a980fd0-c764-477e-b0e8-655fccc406bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:15.788290 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-9a980fd0-c764-477e-b0e8-655fccc406bb None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:42:15.790088 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-9a980fd0-c764-477e-b0e8-655fccc406bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:42:15.857662 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 110/216] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:42:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 09 00:42:15.862741 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5503d7e9-d037-4f7e-804f-ba5cb5d9220c None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:15.878711 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a980fd0-c764-477e-b0e8-655fccc406bb None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:15.920450 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:15.979410 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5503d7e9-d037-4f7e-804f-ba5cb5d9220c None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:15.980216 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a980fd0-c764-477e-b0e8-655fccc406bb None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:15.981989 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:16.014620 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5f1068b-1476-4387-b71e-35df949fe8c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:16.015275 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5f1068b-1476-4387-b71e-35df949fe8c6 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:16.015558 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5f1068b-1476-4387-b71e-35df949fe8c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:16.015906 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5f1068b-1476-4387-b71e-35df949fe8c6 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:16.018359 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29d94365-a2cc-4447-8f13-968a111c9b24 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:16.019107 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29d94365-a2cc-4447-8f13-968a111c9b24 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:16.019393 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29d94365-a2cc-4447-8f13-968a111c9b24 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:16.019702 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29d94365-a2cc-4447-8f13-968a111c9b24 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:16.027205 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5f1068b-1476-4387-b71e-35df949fe8c6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:42:16.031278 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29d94365-a2cc-4447-8f13-968a111c9b24 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:42:16.032250 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-b5f1068b-1476-4387-b71e-35df949fe8c6 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 09 00:42:16.032250 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:42:16.032250 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:42:16.032250 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:42:16.032250 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:42:16.032250 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:42:16.032250 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:42:16.032250 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:42:16.032250 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:42:16.032250 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:42:16.032250 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:42:16.032250 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:42:16.032250 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:42:16.032250 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:42:16.032250 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:42:16.032250 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:42:16.033098 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:42:16.033098 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:42:16.033098 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:42:16.033098 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:42:16.033098 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:42:16.033098 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:42:16.033098 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:42:16.033098 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:42:16.033098 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:42:16.033098 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:42:16.033098 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:42:16.033098 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:42:16.033098 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:42:16.033098 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:42:16.033098 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:42:16.033098 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:42:16.033855 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:42:16.033855 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:42:16.033855 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:42:16.033855 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:42:16.033855 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:42:16.033855 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:42:16.033855 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:42:16.033855 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:42:16.033855 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:42:16.033855 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:42:16.033855 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 09 00:42:16.033855 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:42:16.033855 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 111/217] 10.222.0.33 () {56 vars in 1099 bytes} [Sun Oct 9 00:42:15 2022] GET /identity/v3/roles/member => generated 84 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 09 00:42:16.036334 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-29d94365-a2cc-4447-8f13-968a111c9b24 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 09 00:42:16.036334 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:42:16.036334 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:42:16.036334 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:42:16.036334 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:42:16.036334 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:42:16.036334 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:42:16.036334 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:42:16.036334 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:42:16.036334 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:42:16.036334 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:42:16.036334 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:42:16.036334 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:42:16.036334 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:42:16.036334 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:42:16.036334 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:42:16.037860 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:42:16.037860 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:42:16.037860 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:42:16.037860 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:42:16.037860 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:42:16.037860 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:42:16.037860 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:42:16.037860 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:42:16.037860 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:42:16.037860 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:42:16.037860 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:42:16.037860 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:42:16.037860 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:42:16.037860 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:42:16.037860 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:42:16.037860 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:42:16.039530 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:42:16.039530 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:42:16.039530 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:42:16.039530 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:42:16.039530 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:42:16.039530 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:42:16.039530 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:42:16.039530 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:42:16.039530 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:42:16.039530 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:42:16.039530 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 09 00:42:16.039530 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:42:16.040303 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5f1068b-1476-4387-b71e-35df949fe8c6 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:16.040399 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 107/218] 10.222.0.33 () {56 vars in 1096 bytes} [Sun Oct 9 00:42:15 2022] GET /identity/v3/roles/admin => generated 83 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:42:16.044288 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29d94365-a2cc-4447-8f13-968a111c9b24 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:16.150012 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:16.153394 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5f1068b-1476-4387-b71e-35df949fe8c6 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:16.154397 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29d94365-a2cc-4447-8f13-968a111c9b24 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:16.188096 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d42b06c-e89e-43ac-844b-fd4b3fdb2fe5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:16.188718 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d42b06c-e89e-43ac-844b-fd4b3fdb2fe5 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:16.188899 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8741523a-2265-421d-93bc-3d1bb29765c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:16.189004 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d42b06c-e89e-43ac-844b-fd4b3fdb2fe5 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:16.189251 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d42b06c-e89e-43ac-844b-fd4b3fdb2fe5 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:16.189497 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8741523a-2265-421d-93bc-3d1bb29765c8 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:16.189779 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8741523a-2265-421d-93bc-3d1bb29765c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:16.190045 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8741523a-2265-421d-93bc-3d1bb29765c8 None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:16.190351 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d42b06c-e89e-43ac-844b-fd4b3fdb2fe5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:16.190697 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8741523a-2265-421d-93bc-3d1bb29765c8 None admin] RBAC: Adding query filter params (name=member) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:16.190996 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8741523a-2265-421d-93bc-3d1bb29765c8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:16.191797 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d42b06c-e89e-43ac-844b-fd4b3fdb2fe5 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:16.192378 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8741523a-2265-421d-93bc-3d1bb29765c8 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:16.193349 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 112/219] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:42:16 2022] GET /identity/v3/domains/default => generated 199 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:16.194096 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d42b06c-e89e-43ac-844b-fd4b3fdb2fe5 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:16.203083 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 108/220] 10.222.0.33 () {56 vars in 1101 bytes} [Sun Oct 9 00:42:16 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:16.203783 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8741523a-2265-421d-93bc-3d1bb29765c8 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:16.230905 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:16.292008 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d42b06c-e89e-43ac-844b-fd4b3fdb2fe5 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:16.301798 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8741523a-2265-421d-93bc-3d1bb29765c8 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:16.324560 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1de947a-320a-4c48-a5fb-2d2b1cba37f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:16.325161 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1de947a-320a-4c48-a5fb-2d2b1cba37f9 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:16.325442 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1de947a-320a-4c48-a5fb-2d2b1cba37f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:16.325716 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1de947a-320a-4c48-a5fb-2d2b1cba37f9 None admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:16.326365 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1de947a-320a-4c48-a5fb-2d2b1cba37f9 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:16.326665 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1de947a-320a-4c48-a5fb-2d2b1cba37f9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:16.328060 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1de947a-320a-4c48-a5fb-2d2b1cba37f9 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:16.334874 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45ef10bb-1e1a-4ae3-b53f-478dffeb3208 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:16.335490 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ef10bb-1e1a-4ae3-b53f-478dffeb3208 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:16.335780 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 113/221] 10.222.0.33 () {56 vars in 1099 bytes} [Sun Oct 9 00:42:16 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:16.340452 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ef10bb-1e1a-4ae3-b53f-478dffeb3208 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:16.340452 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ef10bb-1e1a-4ae3-b53f-478dffeb3208 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:16.340614 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1de947a-320a-4c48-a5fb-2d2b1cba37f9 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:16.347649 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45ef10bb-1e1a-4ae3-b53f-478dffeb3208 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:42:16.352841 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-45ef10bb-1e1a-4ae3-b53f-478dffeb3208 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 09 00:42:16.352841 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:42:16.352841 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:42:16.352841 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:42:16.352841 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:42:16.352841 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:42:16.352841 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:42:16.352841 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:42:16.352841 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:42:16.352841 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:42:16.352841 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:42:16.352841 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:42:16.352841 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:42:16.352841 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:42:16.352841 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:42:16.352841 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:42:16.354373 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:42:16.354373 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:42:16.354373 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:42:16.354373 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:42:16.354373 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:42:16.354373 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:42:16.354373 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:42:16.354373 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:42:16.354373 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:42:16.354373 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:42:16.354373 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:42:16.354373 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:42:16.354373 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:42:16.354373 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:42:16.354373 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:42:16.354373 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:42:16.355133 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:42:16.355133 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:42:16.355133 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:42:16.355133 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:42:16.355133 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:42:16.355133 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:42:16.355133 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:42:16.355133 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:42:16.355133 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:42:16.355133 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:42:16.355133 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 09 00:42:16.355133 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:42:16.355133 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 109/222] 10.222.0.33 () {56 vars in 1114 bytes} [Sun Oct 9 00:42:16 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:42:16.355792 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45ef10bb-1e1a-4ae3-b53f-478dffeb3208 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:16.366825 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:16.435121 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1de947a-320a-4c48-a5fb-2d2b1cba37f9 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:16.440104 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:16.454858 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45ef10bb-1e1a-4ae3-b53f-478dffeb3208 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:16.467155 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23acdae3-ff23-45d8-859f-aebe65f888b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:16.467787 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23acdae3-ff23-45d8-859f-aebe65f888b4 None admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:16.468073 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23acdae3-ff23-45d8-859f-aebe65f888b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:16.468350 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23acdae3-ff23-45d8-859f-aebe65f888b4 None admin] PATH_INFO: `/v3/users` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:16.469000 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23acdae3-ff23-45d8-859f-aebe65f888b4 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:16.470427 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23acdae3-ff23-45d8-859f-aebe65f888b4 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:16.488168 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07546b8c-8f28-4fd0-8a58-80b30f90c73d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:16.488781 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07546b8c-8f28-4fd0-8a58-80b30f90c73d None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:16.489037 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07546b8c-8f28-4fd0-8a58-80b30f90c73d None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:16.489325 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07546b8c-8f28-4fd0-8a58-80b30f90c73d None admin] PATH_INFO: `/v3/users/65449b4b768e43ac9da8d8d112d2f701` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:16.491478 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 114/223] 10.222.0.33 () {60 vars in 1132 bytes} [Sun Oct 9 00:42:16 2022] POST /identity/v3/users => generated 297 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:42:16.491478 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23acdae3-ff23-45d8-859f-aebe65f888b4 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:16.509934 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07546b8c-8f28-4fd0-8a58-80b30f90c73d None admin] RBAC: Authorizing `identity:get_user(user_id=65449b4b768e43ac9da8d8d112d2f701)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:16.511428 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07546b8c-8f28-4fd0-8a58-80b30f90c73d None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:16.513406 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 110/224] 10.222.0.33 () {56 vars in 1177 bytes} [Sun Oct 9 00:42:16 2022] GET /identity/v3/users/65449b4b768e43ac9da8d8d112d2f701 => generated 297 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:16.514014 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07546b8c-8f28-4fd0-8a58-80b30f90c73d None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:16.606335 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:16.624439 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07546b8c-8f28-4fd0-8a58-80b30f90c73d None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:16.626176 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23acdae3-ff23-45d8-859f-aebe65f888b4 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:16.662285 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c77ef5a2-b350-400f-9d0a-cf114a651f6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:16.662904 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c77ef5a2-b350-400f-9d0a-cf114a651f6a None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:16.663187 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c77ef5a2-b350-400f-9d0a-cf114a651f6a None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:16.663341 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e9a7e76-8dac-4e94-88e2-a1766130a71f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:16.663527 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c77ef5a2-b350-400f-9d0a-cf114a651f6a None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:16.664265 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e9a7e76-8dac-4e94-88e2-a1766130a71f None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:16.664494 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c77ef5a2-b350-400f-9d0a-cf114a651f6a None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:16.664586 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e9a7e76-8dac-4e94-88e2-a1766130a71f None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:16.664793 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e9a7e76-8dac-4e94-88e2-a1766130a71f None admin] PATH_INFO: `/v3/users/6c8f41499fe54de28740b35e4f20cf88` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:16.664884 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c77ef5a2-b350-400f-9d0a-cf114a651f6a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:16.666790 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c77ef5a2-b350-400f-9d0a-cf114a651f6a None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:16.667284 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e9a7e76-8dac-4e94-88e2-a1766130a71f None admin] RBAC: Authorizing `identity:get_user(user_id=6c8f41499fe54de28740b35e4f20cf88)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:16.668739 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e9a7e76-8dac-4e94-88e2-a1766130a71f None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:16.670456 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 115/225] 10.222.0.33 () {56 vars in 1177 bytes} [Sun Oct 9 00:42:16 2022] GET /identity/v3/users/6c8f41499fe54de28740b35e4f20cf88 => generated 283 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:16.671163 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e9a7e76-8dac-4e94-88e2-a1766130a71f None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:16.676757 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 111/226] 10.222.0.33 () {56 vars in 1111 bytes} [Sun Oct 9 00:42:16 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:16.682524 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c77ef5a2-b350-400f-9d0a-cf114a651f6a None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:16.692858 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:16.801744 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e9a7e76-8dac-4e94-88e2-a1766130a71f None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:16.808908 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c77ef5a2-b350-400f-9d0a-cf114a651f6a None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:16.840077 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da7d6f9c-dc54-4409-95ec-f69a8a17ec39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:16.840778 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da7d6f9c-dc54-4409-95ec-f69a8a17ec39 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:16.841058 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da7d6f9c-dc54-4409-95ec-f69a8a17ec39 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:16.842389 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da7d6f9c-dc54-4409-95ec-f69a8a17ec39 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:16.848751 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ae791b4-5fa7-4664-93d5-32d44f17d688 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:16.849712 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae791b4-5fa7-4664-93d5-32d44f17d688 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:16.850196 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae791b4-5fa7-4664-93d5-32d44f17d688 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:16.850667 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae791b4-5fa7-4664-93d5-32d44f17d688 None admin] PATH_INFO: `/v3/projects/dd8011750cb34075a6f8106533f848c2` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:16.852490 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ae791b4-5fa7-4664-93d5-32d44f17d688 None admin] RBAC: Authorizing `identity:get_project(project_id=dd8011750cb34075a6f8106533f848c2)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:16.855046 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ae791b4-5fa7-4664-93d5-32d44f17d688 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:16.857467 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 112/227] 10.222.0.33 () {56 vars in 1186 bytes} [Sun Oct 9 00:42:16 2022] GET /identity/v3/projects/dd8011750cb34075a6f8106533f848c2 => generated 302 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:16.858842 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ae791b4-5fa7-4664-93d5-32d44f17d688 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:16.870945 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:16.934375 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da7d6f9c-dc54-4409-95ec-f69a8a17ec39 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:16.936846 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da7d6f9c-dc54-4409-95ec-f69a8a17ec39 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:16.945596 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:16.973501 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ae791b4-5fa7-4664-93d5-32d44f17d688 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:17.008122 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d8b7069-cac4-401e-95d6-fda39f538121 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:17.009003 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8b7069-cac4-401e-95d6-fda39f538121 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:17.009429 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8b7069-cac4-401e-95d6-fda39f538121 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:17.010388 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8b7069-cac4-401e-95d6-fda39f538121 None admin] PATH_INFO: `/v3/users/6c8f41499fe54de28740b35e4f20cf88` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:17.011745 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d8b7069-cac4-401e-95d6-fda39f538121 None admin] RBAC: Authorizing `identity:get_user(user_id=6c8f41499fe54de28740b35e4f20cf88)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:17.013356 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d8b7069-cac4-401e-95d6-fda39f538121 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:17.015398 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 113/228] 10.222.0.33 () {56 vars in 1177 bytes} [Sun Oct 9 00:42:16 2022] GET /identity/v3/users/6c8f41499fe54de28740b35e4f20cf88 => generated 283 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:17.016320 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d8b7069-cac4-401e-95d6-fda39f538121 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:17.067967 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 116/229] 10.222.0.33 () {58 vars in 1282 bytes} [Sun Oct 9 00:42:16 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 397 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 09 00:42:17.069206 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da7d6f9c-dc54-4409-95ec-f69a8a17ec39 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:17.136475 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d8b7069-cac4-401e-95d6-fda39f538121 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:17.150490 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:17.173996 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e032ecf5-8bc6-45ba-8820-8077098f8dd0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:17.174827 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e032ecf5-8bc6-45ba-8820-8077098f8dd0 None admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:17.175230 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e032ecf5-8bc6-45ba-8820-8077098f8dd0 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:17.175617 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e032ecf5-8bc6-45ba-8820-8077098f8dd0 None admin] PATH_INFO: `/v3/projects/dd8011750cb34075a6f8106533f848c2/users/6c8f41499fe54de28740b35e4f20cf88/roles/092036ee46ac427baf4d56468575f697` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:17.177951 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e032ecf5-8bc6-45ba-8820-8077098f8dd0 None admin] RBAC: Authorizing `identity:create_grant(project_id=dd8011750cb34075a6f8106533f848c2, user_id=6c8f41499fe54de28740b35e4f20cf88, role_id=092036ee46ac427baf4d56468575f697)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:17.179541 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e032ecf5-8bc6-45ba-8820-8077098f8dd0 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:17.196933 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 114/230] 10.222.0.33 () {58 vars in 1439 bytes} [Sun Oct 9 00:42:17 2022] PUT /identity/v3/projects/dd8011750cb34075a6f8106533f848c2/users/6c8f41499fe54de28740b35e4f20cf88/roles/092036ee46ac427baf4d56468575f697 => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:42:17.198137 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e032ecf5-8bc6-45ba-8820-8077098f8dd0 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:17.201291 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:17.219194 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da7d6f9c-dc54-4409-95ec-f69a8a17ec39 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:17.265179 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8bbe3ba-13d6-4c19-9ea4-ed0b25e03f33 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:17.266057 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8bbe3ba-13d6-4c19-9ea4-ed0b25e03f33 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:17.266517 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8bbe3ba-13d6-4c19-9ea4-ed0b25e03f33 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:17.266974 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8bbe3ba-13d6-4c19-9ea4-ed0b25e03f33 None admin] PATH_INFO: `/v3/projects/dd8011750cb34075a6f8106533f848c2` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:17.268317 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8bbe3ba-13d6-4c19-9ea4-ed0b25e03f33 None admin] RBAC: Authorizing `identity:get_project(project_id=dd8011750cb34075a6f8106533f848c2)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:17.270601 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8bbe3ba-13d6-4c19-9ea4-ed0b25e03f33 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:17.272907 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 117/231] 10.222.0.33 () {56 vars in 1186 bytes} [Sun Oct 9 00:42:17 2022] GET /identity/v3/projects/dd8011750cb34075a6f8106533f848c2 => generated 302 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:17.279454 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8bbe3ba-13d6-4c19-9ea4-ed0b25e03f33 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:17.336199 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e032ecf5-8bc6-45ba-8820-8077098f8dd0 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:17.414307 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a6d5a5-bcf9-451a-bfe9-a49a1a1519bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:17.414944 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a6d5a5-bcf9-451a-bfe9-a49a1a1519bb None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:17.415241 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a6d5a5-bcf9-451a-bfe9-a49a1a1519bb None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:17.415504 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a6d5a5-bcf9-451a-bfe9-a49a1a1519bb None admin] PATH_INFO: `/v3/projects/dd8011750cb34075a6f8106533f848c2` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:17.417257 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94a6d5a5-bcf9-451a-bfe9-a49a1a1519bb None admin] RBAC: Authorizing `identity:get_project(project_id=dd8011750cb34075a6f8106533f848c2)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:17.418763 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94a6d5a5-bcf9-451a-bfe9-a49a1a1519bb None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:17.420771 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 115/232] 10.222.0.33 () {56 vars in 1186 bytes} [Sun Oct 9 00:42:17 2022] GET /identity/v3/projects/dd8011750cb34075a6f8106533f848c2 => generated 302 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:17.426839 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a6d5a5-bcf9-451a-bfe9-a49a1a1519bb None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:17.464067 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8bbe3ba-13d6-4c19-9ea4-ed0b25e03f33 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:17.490695 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:17.504546 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d138f25-d084-42e4-9d49-557800823ff9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:17.505209 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d138f25-d084-42e4-9d49-557800823ff9 None admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:17.505487 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d138f25-d084-42e4-9d49-557800823ff9 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:17.505764 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d138f25-d084-42e4-9d49-557800823ff9 None admin] PATH_INFO: `/v3/projects/dd8011750cb34075a6f8106533f848c2/users/6c8f41499fe54de28740b35e4f20cf88/roles/7c1fdac272a34be99661077b0f1b8014` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:17.508126 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d138f25-d084-42e4-9d49-557800823ff9 None admin] RBAC: Authorizing `identity:create_grant(project_id=dd8011750cb34075a6f8106533f848c2, user_id=6c8f41499fe54de28740b35e4f20cf88, role_id=7c1fdac272a34be99661077b0f1b8014)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:17.509311 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:17.511940 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d138f25-d084-42e4-9d49-557800823ff9 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:17.532716 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 118/233] 10.222.0.33 () {58 vars in 1439 bytes} [Sun Oct 9 00:42:17 2022] PUT /identity/v3/projects/dd8011750cb34075a6f8106533f848c2/users/6c8f41499fe54de28740b35e4f20cf88/roles/7c1fdac272a34be99661077b0f1b8014 => generated 0 bytes in 253 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:42:17.556290 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a6d5a5-bcf9-451a-bfe9-a49a1a1519bb None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:17.598594 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bf3d2cf-1b40-4489-b4ca-bae96334913e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:17.599227 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bf3d2cf-1b40-4489-b4ca-bae96334913e None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:17.599506 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bf3d2cf-1b40-4489-b4ca-bae96334913e None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:17.599855 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bf3d2cf-1b40-4489-b4ca-bae96334913e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:17.600828 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bf3d2cf-1b40-4489-b4ca-bae96334913e None admin] RBAC: Adding query filter params (role.id=588016933bab436d8b7c3ac76827356d, scope.project.id=dd8011750cb34075a6f8106533f848c2, user.id=65449b4b768e43ac9da8d8d112d2f701) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:17.601143 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bf3d2cf-1b40-4489-b4ca-bae96334913e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:17.602775 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bf3d2cf-1b40-4489-b4ca-bae96334913e None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:17.610352 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 116/234] 10.222.0.33 () {56 vars in 1374 bytes} [Sun Oct 9 00:42:17 2022] GET /identity/v3/role_assignments?user.id=65449b4b768e43ac9da8d8d112d2f701&scope.project.id=dd8011750cb34075a6f8106533f848c2&role.id=588016933bab436d8b7c3ac76827356d => generated 259 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:42:19.129551 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6769bf85-aaef-4a2f-a2e2-750a551a77ba None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:19.130453 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6769bf85-aaef-4a2f-a2e2-750a551a77ba None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:19.130906 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6769bf85-aaef-4a2f-a2e2-750a551a77ba None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:19.132455 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 119/235] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:42:19 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:42:19.137825 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e2c3b10-864a-4d58-8840-2982c14a54b1 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:19.138629 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e2c3b10-864a-4d58-8840-2982c14a54b1 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:19.139078 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e2c3b10-864a-4d58-8840-2982c14a54b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:19.168814 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-0e2c3b10-864a-4d58-8840-2982c14a54b1 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:42:19.170320 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-0e2c3b10-864a-4d58-8840-2982c14a54b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:42:19.185164 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f28b95-e47a-452b-9700-644e96ac47e7 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:19.185727 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f28b95-e47a-452b-9700-644e96ac47e7 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:19.186172 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f28b95-e47a-452b-9700-644e96ac47e7 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:19.187937 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 120/236] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:42:19 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:42:19.192716 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8984e5c6-ec36-476a-990f-14b4fd027be1 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:19.193245 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8984e5c6-ec36-476a-990f-14b4fd027be1 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:19.193745 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8984e5c6-ec36-476a-990f-14b4fd027be1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:19.225227 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-8984e5c6-ec36-476a-990f-14b4fd027be1 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:42:19.226813 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-8984e5c6-ec36-476a-990f-14b4fd027be1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:42:19.248837 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 117/237] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:42:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 09 00:42:19.340036 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 121/238] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:42:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 09 00:42:19.638515 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e2c3b10-864a-4d58-8840-2982c14a54b1 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:19.648642 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8984e5c6-ec36-476a-990f-14b4fd027be1 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:19.744802 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:19.797076 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e2c3b10-864a-4d58-8840-2982c14a54b1 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:19.818433 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8984e5c6-ec36-476a-990f-14b4fd027be1 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:19.833168 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:19.860049 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d39e3f1-18ad-43ce-9df8-ead210275671 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:19.861092 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d39e3f1-18ad-43ce-9df8-ead210275671 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:19.861552 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d39e3f1-18ad-43ce-9df8-ead210275671 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:19.862327 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d39e3f1-18ad-43ce-9df8-ead210275671 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:19.865657 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d39e3f1-18ad-43ce-9df8-ead210275671 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:19.868885 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d39e3f1-18ad-43ce-9df8-ead210275671 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:19.873438 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 118/239] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:42:19 2022] GET /identity/v3/domains/default => generated 199 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:19.880173 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d39e3f1-18ad-43ce-9df8-ead210275671 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:19.887107 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-964fbc17-4afd-4bea-883b-676ab246bf59 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:19.888089 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-964fbc17-4afd-4bea-883b-676ab246bf59 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:19.888633 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-964fbc17-4afd-4bea-883b-676ab246bf59 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:19.889081 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-964fbc17-4afd-4bea-883b-676ab246bf59 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:19.916683 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-964fbc17-4afd-4bea-883b-676ab246bf59 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:42:19.921510 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-964fbc17-4afd-4bea-883b-676ab246bf59 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 09 00:42:19.921510 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:42:19.921510 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:42:19.921510 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:42:19.921510 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:42:19.921510 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:42:19.921510 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:42:19.921510 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:42:19.921510 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:42:19.921510 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:42:19.921510 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:42:19.921510 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:42:19.921510 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:42:19.921510 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:42:19.921510 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:42:19.921510 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:42:19.922358 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:42:19.922358 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:42:19.922358 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:42:19.922358 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:42:19.922358 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:42:19.922358 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:42:19.922358 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:42:19.922358 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:42:19.922358 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:42:19.922358 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:42:19.922358 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:42:19.922358 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:42:19.922358 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:42:19.922358 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:42:19.922358 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:42:19.922358 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:42:19.923125 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:42:19.923125 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:42:19.923125 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:42:19.923125 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:42:19.923125 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:42:19.923125 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:42:19.923125 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:42:19.923125 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:42:19.923125 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:42:19.923125 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:42:19.923125 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 09 00:42:19.923125 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:42:19.924517 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 122/240] 10.222.0.33 () {56 vars in 1099 bytes} [Sun Oct 9 00:42:19 2022] GET /identity/v3/roles/reader => generated 84 bytes in 276 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:42:19.930346 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-964fbc17-4afd-4bea-883b-676ab246bf59 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:20.006447 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d39e3f1-18ad-43ce-9df8-ead210275671 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:20.044032 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:20.056608 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-927b4840-5c25-4160-a608-5e50aa8187ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:20.058660 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-927b4840-5c25-4160-a608-5e50aa8187ea None admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:20.059832 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-927b4840-5c25-4160-a608-5e50aa8187ea None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:20.060890 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-927b4840-5c25-4160-a608-5e50aa8187ea None admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:20.061869 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-927b4840-5c25-4160-a608-5e50aa8187ea None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:20.065055 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-927b4840-5c25-4160-a608-5e50aa8187ea None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:20.076053 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-964fbc17-4afd-4bea-883b-676ab246bf59 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:20.111229 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 119/241] 10.222.0.33 () {60 vars in 1132 bytes} [Sun Oct 9 00:42:19 2022] POST /identity/v3/users => generated 293 bytes in 231 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:42:20.112851 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec15a1c8-5b30-4cbc-ae21-9e84410ec901 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:20.113635 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec15a1c8-5b30-4cbc-ae21-9e84410ec901 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:20.113635 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec15a1c8-5b30-4cbc-ae21-9e84410ec901 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:20.116840 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 120/242] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:42:20 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:42:20.127628 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df19eee-6ce1-4add-a363-c6da2ad52654 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:20.127628 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df19eee-6ce1-4add-a363-c6da2ad52654 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:20.127628 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df19eee-6ce1-4add-a363-c6da2ad52654 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:20.155713 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdf42aff-b36b-46a5-9fb4-f6de4a8b519b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:20.155713 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf42aff-b36b-46a5-9fb4-f6de4a8b519b None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:20.155713 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf42aff-b36b-46a5-9fb4-f6de4a8b519b None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:20.159613 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf42aff-b36b-46a5-9fb4-f6de4a8b519b None admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:20.159613 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdf42aff-b36b-46a5-9fb4-f6de4a8b519b None admin] RBAC: Adding query filter params (name=reader) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:20.163617 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdf42aff-b36b-46a5-9fb4-f6de4a8b519b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:20.164005 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-6df19eee-6ce1-4add-a363-c6da2ad52654 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:42:20.164005 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-6df19eee-6ce1-4add-a363-c6da2ad52654 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:42:20.164444 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdf42aff-b36b-46a5-9fb4-f6de4a8b519b None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:20.187617 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 123/243] 10.222.0.33 () {56 vars in 1101 bytes} [Sun Oct 9 00:42:19 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 256 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:20.199625 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdf42aff-b36b-46a5-9fb4-f6de4a8b519b None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:20.271620 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:20.309792 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 121/244] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:42:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 09 00:42:20.339513 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a755cb2-757e-4c57-9cfd-f6de5654f4c6 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:20.339871 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a755cb2-757e-4c57-9cfd-f6de5654f4c6 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:20.340111 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a755cb2-757e-4c57-9cfd-f6de5654f4c6 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:20.341552 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 122/245] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:42:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:42:20.350769 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-024b827d-810c-4f8c-9fff-2a783e0045ec None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:20.351083 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-024b827d-810c-4f8c-9fff-2a783e0045ec None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:20.351383 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-024b827d-810c-4f8c-9fff-2a783e0045ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:20.386829 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-024b827d-810c-4f8c-9fff-2a783e0045ec None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:42:20.388098 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-024b827d-810c-4f8c-9fff-2a783e0045ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:42:20.421065 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdf42aff-b36b-46a5-9fb4-f6de4a8b519b None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:20.461072 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42b6d98c-3282-441a-9917-1ffedd352c6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:20.461710 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42b6d98c-3282-441a-9917-1ffedd352c6a None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:20.461994 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42b6d98c-3282-441a-9917-1ffedd352c6a None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:20.462259 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42b6d98c-3282-441a-9917-1ffedd352c6a None admin] PATH_INFO: `/v3/users/13e948e5692e4a7ab9711a3062b73ad7` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:20.477487 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42b6d98c-3282-441a-9917-1ffedd352c6a None admin] RBAC: Authorizing `identity:get_user(user_id=13e948e5692e4a7ab9711a3062b73ad7)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:20.479905 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42b6d98c-3282-441a-9917-1ffedd352c6a None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:20.481760 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 124/246] 10.222.0.33 () {56 vars in 1177 bytes} [Sun Oct 9 00:42:20 2022] GET /identity/v3/users/13e948e5692e4a7ab9711a3062b73ad7 => generated 297 bytes in 285 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:42:20.482796 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e2419a3-a61f-4080-b364-ad4ed284d957 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:20.483081 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e2419a3-a61f-4080-b364-ad4ed284d957 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:20.483353 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e2419a3-a61f-4080-b364-ad4ed284d957 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:20.489286 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 125/247] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:42:20 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:42:20.490063 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e2419a3-a61f-4080-b364-ad4ed284d957 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:20.551618 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 123/248] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:42:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 195 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 09 00:42:20.551618 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db73b264-a623-4520-98ac-62293ea9ca5f None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:20.551618 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db73b264-a623-4520-98ac-62293ea9ca5f None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:20.551618 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db73b264-a623-4520-98ac-62293ea9ca5f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:20.573767 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-db73b264-a623-4520-98ac-62293ea9ca5f None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:42:20.575747 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-db73b264-a623-4520-98ac-62293ea9ca5f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:42:20.630257 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:20.659220 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e2419a3-a61f-4080-b364-ad4ed284d957 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:20.670307 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:20.703487 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 124/249] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:42:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 09 00:42:20.705572 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db73b264-a623-4520-98ac-62293ea9ca5f None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:20.724323 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7f99375-f627-4157-a03b-44a675cd9f3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:20.724949 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7f99375-f627-4157-a03b-44a675cd9f3d None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:20.725230 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7f99375-f627-4157-a03b-44a675cd9f3d None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:20.725502 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7f99375-f627-4157-a03b-44a675cd9f3d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:20.826511 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7f99375-f627-4157-a03b-44a675cd9f3d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:20.828009 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7f99375-f627-4157-a03b-44a675cd9f3d None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:20.837257 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db73b264-a623-4520-98ac-62293ea9ca5f None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:20.878246 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f7a71c5-6b51-4160-9e2b-73fa3ed7b9e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:20.878926 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f7a71c5-6b51-4160-9e2b-73fa3ed7b9e9 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:20.879206 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f7a71c5-6b51-4160-9e2b-73fa3ed7b9e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:20.879478 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f7a71c5-6b51-4160-9e2b-73fa3ed7b9e9 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:20.894926 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f7a71c5-6b51-4160-9e2b-73fa3ed7b9e9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:42:20.901266 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-5f7a71c5-6b51-4160-9e2b-73fa3ed7b9e9 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 09 00:42:20.901266 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:42:20.901266 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:42:20.901266 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:42:20.901266 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:42:20.901266 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:42:20.901266 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:42:20.901266 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:42:20.901266 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:42:20.901266 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:42:20.901266 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:42:20.901266 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:42:20.901266 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:42:20.901266 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:42:20.901266 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:42:20.901266 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:42:20.902122 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:42:20.902122 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:42:20.902122 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:42:20.902122 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:42:20.902122 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:42:20.902122 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:42:20.902122 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:42:20.902122 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:42:20.902122 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:42:20.902122 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:42:20.902122 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:42:20.902122 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:42:20.902122 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:42:20.902122 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:42:20.902122 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:42:20.902122 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:42:20.902901 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:42:20.902901 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:42:20.902901 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:42:20.902901 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:42:20.902901 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:42:20.902901 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:42:20.902901 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:42:20.902901 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:42:20.902901 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:42:20.902901 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:42:20.902901 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 09 00:42:20.902901 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:42:20.905649 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 125/250] 10.222.0.33 () {56 vars in 1096 bytes} [Sun Oct 9 00:42:20 2022] GET /identity/v3/roles/admin => generated 83 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:42:20.906720 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f7a71c5-6b51-4160-9e2b-73fa3ed7b9e9 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:20.934355 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:20.946686 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 126/251] 10.222.0.33 () {58 vars in 1282 bytes} [Sun Oct 9 00:42:20 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 457 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 09 00:42:20.948304 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7f99375-f627-4157-a03b-44a675cd9f3d None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:20.958830 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:21.029325 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f7a71c5-6b51-4160-9e2b-73fa3ed7b9e9 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:21.070257 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48305ad7-6ada-42da-85a2-b06bd33a05de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:21.071218 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7f99375-f627-4157-a03b-44a675cd9f3d None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:21.071500 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48305ad7-6ada-42da-85a2-b06bd33a05de None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:21.072040 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48305ad7-6ada-42da-85a2-b06bd33a05de None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:21.072505 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48305ad7-6ada-42da-85a2-b06bd33a05de None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:21.091626 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48305ad7-6ada-42da-85a2-b06bd33a05de None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:42:21.093878 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-48305ad7-6ada-42da-85a2-b06bd33a05de None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 09 00:42:21.093878 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:42:21.093878 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:42:21.093878 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:42:21.093878 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:42:21.093878 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:42:21.093878 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:42:21.093878 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:42:21.093878 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:42:21.093878 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:42:21.093878 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:42:21.093878 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:42:21.093878 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:42:21.093878 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:42:21.093878 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:42:21.093878 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:42:21.094707 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:42:21.094707 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:42:21.094707 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:42:21.094707 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:42:21.094707 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:42:21.094707 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:42:21.094707 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:42:21.094707 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:42:21.094707 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:42:21.094707 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:42:21.094707 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:42:21.094707 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:42:21.094707 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:42:21.094707 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:42:21.094707 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:42:21.094707 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:42:21.095460 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:42:21.095460 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:42:21.095460 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:42:21.095460 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:42:21.095460 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:42:21.095460 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:42:21.095460 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:42:21.095460 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:42:21.095460 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:42:21.095460 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:42:21.095460 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 09 00:42:21.095460 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:42:21.096068 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 126/252] 10.222.0.33 () {56 vars in 1114 bytes} [Sun Oct 9 00:42:20 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:42:21.096787 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48305ad7-6ada-42da-85a2-b06bd33a05de None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:21.113151 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28d20857-febf-4ab9-8f2d-9245b140fd49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:21.114094 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d20857-febf-4ab9-8f2d-9245b140fd49 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:21.114574 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d20857-febf-4ab9-8f2d-9245b140fd49 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:21.115057 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d20857-febf-4ab9-8f2d-9245b140fd49 None admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:21.115957 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28d20857-febf-4ab9-8f2d-9245b140fd49 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:21.116475 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28d20857-febf-4ab9-8f2d-9245b140fd49 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:21.118604 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28d20857-febf-4ab9-8f2d-9245b140fd49 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:21.128947 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 127/253] 10.222.0.33 () {56 vars in 1099 bytes} [Sun Oct 9 00:42:20 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:21.130144 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28d20857-febf-4ab9-8f2d-9245b140fd49 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:21.195954 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:21.220103 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:21.227044 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48305ad7-6ada-42da-85a2-b06bd33a05de None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:21.250558 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28d20857-febf-4ab9-8f2d-9245b140fd49 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:21.276093 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4af567aa-0c27-45d5-b361-0217bf2be840 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:21.277329 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af567aa-0c27-45d5-b361-0217bf2be840 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:21.277708 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af567aa-0c27-45d5-b361-0217bf2be840 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:21.278055 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af567aa-0c27-45d5-b361-0217bf2be840 None admin] PATH_INFO: `/v3/projects/dd8011750cb34075a6f8106533f848c2` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:21.279440 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4af567aa-0c27-45d5-b361-0217bf2be840 None admin] RBAC: Authorizing `identity:get_project(project_id=dd8011750cb34075a6f8106533f848c2)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:21.281652 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4af567aa-0c27-45d5-b361-0217bf2be840 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:21.283896 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 127/254] 10.222.0.33 () {56 vars in 1186 bytes} [Sun Oct 9 00:42:21 2022] GET /identity/v3/projects/dd8011750cb34075a6f8106533f848c2 => generated 302 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:21.289255 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4af567aa-0c27-45d5-b361-0217bf2be840 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:21.297828 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-745a47a9-dd1c-49c4-8ec7-636a0a7689fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:21.298467 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-745a47a9-dd1c-49c4-8ec7-636a0a7689fe None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:21.298745 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-745a47a9-dd1c-49c4-8ec7-636a0a7689fe None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:21.299015 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-745a47a9-dd1c-49c4-8ec7-636a0a7689fe None admin] PATH_INFO: `/v3/roles/member` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:21.313515 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-745a47a9-dd1c-49c4-8ec7-636a0a7689fe None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:42:21.321420 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-745a47a9-dd1c-49c4-8ec7-636a0a7689fe None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 09 00:42:21.321420 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:42:21.321420 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:42:21.321420 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:42:21.321420 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:42:21.321420 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:42:21.321420 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:42:21.321420 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:42:21.321420 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:42:21.321420 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:42:21.321420 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:42:21.321420 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:42:21.321420 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:42:21.321420 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:42:21.321420 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:42:21.321420 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:42:21.322878 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:42:21.322878 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:42:21.322878 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:42:21.322878 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:42:21.322878 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:42:21.322878 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:42:21.322878 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:42:21.322878 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:42:21.322878 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:42:21.322878 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:42:21.322878 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:42:21.322878 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:42:21.322878 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:42:21.322878 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:42:21.322878 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:42:21.322878 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:42:21.324057 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:42:21.324057 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:42:21.324057 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:42:21.324057 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:42:21.324057 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:42:21.324057 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:42:21.324057 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:42:21.324057 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:42:21.324057 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:42:21.324057 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:42:21.324057 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 09 00:42:21.324057 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:42:21.324057 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 128/255] 10.222.0.33 () {56 vars in 1099 bytes} [Sun Oct 9 00:42:21 2022] GET /identity/v3/roles/member => generated 84 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:42:21.324057 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-745a47a9-dd1c-49c4-8ec7-636a0a7689fe None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:21.400609 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4af567aa-0c27-45d5-b361-0217bf2be840 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:21.444764 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80e84740-dfef-4b72-92b4-370df74ebb73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:21.445413 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e84740-dfef-4b72-92b4-370df74ebb73 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:21.445696 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e84740-dfef-4b72-92b4-370df74ebb73 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:21.446020 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e84740-dfef-4b72-92b4-370df74ebb73 None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:21.446756 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80e84740-dfef-4b72-92b4-370df74ebb73 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:21.447096 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80e84740-dfef-4b72-92b4-370df74ebb73 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:21.449203 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80e84740-dfef-4b72-92b4-370df74ebb73 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:21.461303 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 128/256] 10.222.0.33 () {56 vars in 1111 bytes} [Sun Oct 9 00:42:21 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:21.463024 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80e84740-dfef-4b72-92b4-370df74ebb73 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:21.468207 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-745a47a9-dd1c-49c4-8ec7-636a0a7689fe None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:21.485317 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:21.505148 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:21.507700 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46b20727-3283-4549-a936-c826540ed63c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:21.508582 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b20727-3283-4549-a936-c826540ed63c None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:21.509083 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b20727-3283-4549-a936-c826540ed63c None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:21.509685 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b20727-3283-4549-a936-c826540ed63c None admin] PATH_INFO: `/v3/users/6c8f41499fe54de28740b35e4f20cf88` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:21.511073 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46b20727-3283-4549-a936-c826540ed63c None admin] RBAC: Authorizing `identity:get_user(user_id=6c8f41499fe54de28740b35e4f20cf88)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:21.512837 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46b20727-3283-4549-a936-c826540ed63c None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:21.514643 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 129/257] 10.222.0.33 () {56 vars in 1177 bytes} [Sun Oct 9 00:42:21 2022] GET /identity/v3/users/6c8f41499fe54de28740b35e4f20cf88 => generated 283 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:21.515620 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46b20727-3283-4549-a936-c826540ed63c None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:21.590573 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80e84740-dfef-4b72-92b4-370df74ebb73 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:21.637787 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-115495cf-3b04-4145-b45c-8311d3292db9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:21.638406 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-115495cf-3b04-4145-b45c-8311d3292db9 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:21.638685 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-115495cf-3b04-4145-b45c-8311d3292db9 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:21.638958 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-115495cf-3b04-4145-b45c-8311d3292db9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:21.640002 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-115495cf-3b04-4145-b45c-8311d3292db9 None admin] RBAC: Adding query filter params (role.id=2d631b6eec1646c9834f74efa1d66091, scope.project.id=dd8011750cb34075a6f8106533f848c2, user.id=13e948e5692e4a7ab9711a3062b73ad7) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:21.640316 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-115495cf-3b04-4145-b45c-8311d3292db9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:21.642421 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-115495cf-3b04-4145-b45c-8311d3292db9 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:21.644945 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46b20727-3283-4549-a936-c826540ed63c None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:21.655644 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 129/258] 10.222.0.33 () {56 vars in 1374 bytes} [Sun Oct 9 00:42:21 2022] GET /identity/v3/role_assignments?user.id=13e948e5692e4a7ab9711a3062b73ad7&scope.project.id=dd8011750cb34075a6f8106533f848c2&role.id=2d631b6eec1646c9834f74efa1d66091 => generated 259 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:21.655644 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-115495cf-3b04-4145-b45c-8311d3292db9 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:21.733024 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83c24874-fbee-467c-9338-8131744c5f56 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:21.734971 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c24874-fbee-467c-9338-8131744c5f56 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:21.735562 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c24874-fbee-467c-9338-8131744c5f56 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:21.737284 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c24874-fbee-467c-9338-8131744c5f56 None admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:21.738157 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83c24874-fbee-467c-9338-8131744c5f56 None admin] RBAC: Adding query filter params (name=member) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:21.741045 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83c24874-fbee-467c-9338-8131744c5f56 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:21.743259 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83c24874-fbee-467c-9338-8131744c5f56 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:21.764979 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 130/259] 10.222.0.33 () {56 vars in 1101 bytes} [Sun Oct 9 00:42:21 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:21.768196 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83c24874-fbee-467c-9338-8131744c5f56 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:21.830381 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:21.876638 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:21.892299 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-115495cf-3b04-4145-b45c-8311d3292db9 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:21.924420 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83c24874-fbee-467c-9338-8131744c5f56 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:21.961476 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae773319-de9c-471b-82ad-c421942ddcdb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:21.962094 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae773319-de9c-471b-82ad-c421942ddcdb None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:21.962374 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae773319-de9c-471b-82ad-c421942ddcdb None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:21.963186 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae773319-de9c-471b-82ad-c421942ddcdb None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:21.987374 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56015a71-d619-4b14-b956-3d297d7de10c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:21.987959 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56015a71-d619-4b14-b956-3d297d7de10c None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:21.988238 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56015a71-d619-4b14-b956-3d297d7de10c None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:21.988510 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56015a71-d619-4b14-b956-3d297d7de10c None admin] PATH_INFO: `/v3/users/6c8f41499fe54de28740b35e4f20cf88` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:21.989855 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56015a71-d619-4b14-b956-3d297d7de10c None admin] RBAC: Authorizing `identity:get_user(user_id=6c8f41499fe54de28740b35e4f20cf88)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:21.991316 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56015a71-d619-4b14-b956-3d297d7de10c None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:21.993115 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 130/260] 10.222.0.33 () {56 vars in 1177 bytes} [Sun Oct 9 00:42:21 2022] GET /identity/v3/users/6c8f41499fe54de28740b35e4f20cf88 => generated 283 bytes in 343 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:21.993745 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56015a71-d619-4b14-b956-3d297d7de10c None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:22.075395 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae773319-de9c-471b-82ad-c421942ddcdb None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:22.076909 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae773319-de9c-471b-82ad-c421942ddcdb None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:22.185483 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56015a71-d619-4b14-b956-3d297d7de10c None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:22.191931 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:22.231079 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1689923c-c8ad-4588-bba3-6565218cffba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:22.231773 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1689923c-c8ad-4588-bba3-6565218cffba None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:22.232015 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1689923c-c8ad-4588-bba3-6565218cffba None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:22.232289 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1689923c-c8ad-4588-bba3-6565218cffba None admin] PATH_INFO: `/v3/users/65449b4b768e43ac9da8d8d112d2f701` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:22.236496 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1689923c-c8ad-4588-bba3-6565218cffba None admin] RBAC: Authorizing `identity:get_user(user_id=65449b4b768e43ac9da8d8d112d2f701)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:22.238007 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1689923c-c8ad-4588-bba3-6565218cffba None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:22.239710 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 131/261] 10.222.0.33 () {56 vars in 1177 bytes} [Sun Oct 9 00:42:21 2022] GET /identity/v3/users/65449b4b768e43ac9da8d8d112d2f701 => generated 297 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:22.240299 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1689923c-c8ad-4588-bba3-6565218cffba None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:22.302582 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 131/262] 10.222.0.33 () {58 vars in 1282 bytes} [Sun Oct 9 00:42:21 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 535 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 09 00:42:22.303262 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae773319-de9c-471b-82ad-c421942ddcdb None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:22.303580 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:22.432350 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae773319-de9c-471b-82ad-c421942ddcdb None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:22.469384 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ed47e5-c18f-456e-aec1-fc7d3db108bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:22.470022 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ed47e5-c18f-456e-aec1-fc7d3db108bd None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:22.470303 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ed47e5-c18f-456e-aec1-fc7d3db108bd None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:22.470573 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ed47e5-c18f-456e-aec1-fc7d3db108bd None admin] PATH_INFO: `/v3/projects/dd8011750cb34075a6f8106533f848c2` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:22.471748 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71ed47e5-c18f-456e-aec1-fc7d3db108bd None admin] RBAC: Authorizing `identity:get_project(project_id=dd8011750cb34075a6f8106533f848c2)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:22.473380 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71ed47e5-c18f-456e-aec1-fc7d3db108bd None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:22.475176 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 132/263] 10.222.0.33 () {56 vars in 1186 bytes} [Sun Oct 9 00:42:22 2022] GET /identity/v3/projects/dd8011750cb34075a6f8106533f848c2 => generated 302 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:22.475923 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ed47e5-c18f-456e-aec1-fc7d3db108bd None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:22.505597 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1689923c-c8ad-4588-bba3-6565218cffba None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:22.540014 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:22.578935 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f821e59-7e7f-4d98-88e1-08c240aa28e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:22.579714 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f821e59-7e7f-4d98-88e1-08c240aa28e0 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:22.580001 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f821e59-7e7f-4d98-88e1-08c240aa28e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:22.580401 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f821e59-7e7f-4d98-88e1-08c240aa28e0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:22.614505 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ed47e5-c18f-456e-aec1-fc7d3db108bd None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:22.657377 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f821e59-7e7f-4d98-88e1-08c240aa28e0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:22.658918 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f821e59-7e7f-4d98-88e1-08c240aa28e0 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:22.692671 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:22.698582 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f15ebf5b-91bb-42f2-b578-0517f87afccb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:22.699223 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f15ebf5b-91bb-42f2-b578-0517f87afccb None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:22.699502 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f15ebf5b-91bb-42f2-b578-0517f87afccb None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:22.699801 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f15ebf5b-91bb-42f2-b578-0517f87afccb None admin] PATH_INFO: `/v3/projects/dd8011750cb34075a6f8106533f848c2` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:22.703721 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f15ebf5b-91bb-42f2-b578-0517f87afccb None admin] RBAC: Authorizing `identity:get_project(project_id=dd8011750cb34075a6f8106533f848c2)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:22.705187 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f15ebf5b-91bb-42f2-b578-0517f87afccb None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:22.706826 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 133/264] 10.222.0.33 () {56 vars in 1186 bytes} [Sun Oct 9 00:42:22 2022] GET /identity/v3/projects/dd8011750cb34075a6f8106533f848c2 => generated 302 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:22.707566 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f15ebf5b-91bb-42f2-b578-0517f87afccb None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:22.833386 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 132/265] 10.222.0.33 () {58 vars in 1282 bytes} [Sun Oct 9 00:42:22 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 593 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 09 00:42:22.833993 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f821e59-7e7f-4d98-88e1-08c240aa28e0 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:22.869287 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:22.914268 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f15ebf5b-91bb-42f2-b578-0517f87afccb None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:22.985683 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-150d98d8-09a4-4a07-8d4e-b385104d0c94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:22.986360 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-150d98d8-09a4-4a07-8d4e-b385104d0c94 None admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:22.986645 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-150d98d8-09a4-4a07-8d4e-b385104d0c94 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:22.986924 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-150d98d8-09a4-4a07-8d4e-b385104d0c94 None admin] PATH_INFO: `/v3/projects/dd8011750cb34075a6f8106533f848c2/users/65449b4b768e43ac9da8d8d112d2f701/roles/588016933bab436d8b7c3ac76827356d` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:22.989481 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f821e59-7e7f-4d98-88e1-08c240aa28e0 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:22.993324 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-150d98d8-09a4-4a07-8d4e-b385104d0c94 None admin] RBAC: Authorizing `identity:create_grant(project_id=dd8011750cb34075a6f8106533f848c2, user_id=65449b4b768e43ac9da8d8d112d2f701, role_id=588016933bab436d8b7c3ac76827356d)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:22.994875 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-150d98d8-09a4-4a07-8d4e-b385104d0c94 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:23.019746 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:23.021918 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 134/266] 10.222.0.33 () {58 vars in 1439 bytes} [Sun Oct 9 00:42:22 2022] PUT /identity/v3/projects/dd8011750cb34075a6f8106533f848c2/users/65449b4b768e43ac9da8d8d112d2f701/roles/588016933bab436d8b7c3ac76827356d => generated 0 bytes in 314 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:42:23.022628 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-150d98d8-09a4-4a07-8d4e-b385104d0c94 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:23.051146 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc35976-39f2-4a75-a0e7-3a6f4c806680 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:23.051582 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc35976-39f2-4a75-a0e7-3a6f4c806680 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:23.051884 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc35976-39f2-4a75-a0e7-3a6f4c806680 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:23.052156 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc35976-39f2-4a75-a0e7-3a6f4c806680 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:23.052913 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbc35976-39f2-4a75-a0e7-3a6f4c806680 None admin] RBAC: Adding query filter params (role.id=092036ee46ac427baf4d56468575f697, scope.project.id=dd8011750cb34075a6f8106533f848c2, user.id=6c8f41499fe54de28740b35e4f20cf88) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:23.053315 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbc35976-39f2-4a75-a0e7-3a6f4c806680 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:23.054779 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbc35976-39f2-4a75-a0e7-3a6f4c806680 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:23.072479 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 133/267] 10.222.0.33 () {56 vars in 1374 bytes} [Sun Oct 9 00:42:22 2022] GET /identity/v3/role_assignments?user.id=6c8f41499fe54de28740b35e4f20cf88&scope.project.id=dd8011750cb34075a6f8106533f848c2&role.id=092036ee46ac427baf4d56468575f697 => generated 608 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:23.073847 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0db264-bc30-450f-bb9c-d9454bfb3a1b None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:23.074144 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0db264-bc30-450f-bb9c-d9454bfb3a1b None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:23.074876 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0db264-bc30-450f-bb9c-d9454bfb3a1b None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:23.084158 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 134/268] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:42:23 2022] GET /identity => generated 270 bytes in 11 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:42:23.090530 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bbfe0b5-d3c7-4034-8d4d-e886537e8163 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:23.090838 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bbfe0b5-d3c7-4034-8d4d-e886537e8163 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:23.091092 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bbfe0b5-d3c7-4034-8d4d-e886537e8163 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:23.131353 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-1bbfe0b5-d3c7-4034-8d4d-e886537e8163 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:42:23.132701 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-1bbfe0b5-d3c7-4034-8d4d-e886537e8163 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:42:23.155135 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-150d98d8-09a4-4a07-8d4e-b385104d0c94 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:23.198258 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58bc377e-8dea-48af-86bf-62da5cd09624 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:23.199292 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58bc377e-8dea-48af-86bf-62da5cd09624 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:23.199821 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58bc377e-8dea-48af-86bf-62da5cd09624 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:23.200296 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58bc377e-8dea-48af-86bf-62da5cd09624 None admin] PATH_INFO: `/v3/projects/dd8011750cb34075a6f8106533f848c2` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:23.202351 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58bc377e-8dea-48af-86bf-62da5cd09624 None admin] RBAC: Authorizing `identity:get_project(project_id=dd8011750cb34075a6f8106533f848c2)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:23.204510 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58bc377e-8dea-48af-86bf-62da5cd09624 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:23.206798 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 135/269] 10.222.0.33 () {56 vars in 1186 bytes} [Sun Oct 9 00:42:23 2022] GET /identity/v3/projects/dd8011750cb34075a6f8106533f848c2 => generated 302 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:23.213099 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58bc377e-8dea-48af-86bf-62da5cd09624 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:23.232314 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:23.281831 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 135/270] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:42:23 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 09 00:42:23.284947 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0b576d6-e737-4a2b-8328-d435e4a8096e None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:23.287015 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0b576d6-e737-4a2b-8328-d435e4a8096e None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:23.289795 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0b576d6-e737-4a2b-8328-d435e4a8096e None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:23.291990 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 136/271] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:42:23 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:42:23.298203 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9800db2-86e1-40c4-847b-38e21d5163f3 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:23.298763 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9800db2-86e1-40c4-847b-38e21d5163f3 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:23.299251 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9800db2-86e1-40c4-847b-38e21d5163f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:23.338086 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-e9800db2-86e1-40c4-847b-38e21d5163f3 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:42:23.341129 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-e9800db2-86e1-40c4-847b-38e21d5163f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:42:23.414492 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58bc377e-8dea-48af-86bf-62da5cd09624 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:23.443492 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 137/272] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:42:23 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 09 00:42:23.467957 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77f7b234-10af-422c-91a6-7486ba39fe22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:23.468625 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f7b234-10af-422c-91a6-7486ba39fe22 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:23.468910 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f7b234-10af-422c-91a6-7486ba39fe22 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:23.469190 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f7b234-10af-422c-91a6-7486ba39fe22 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:23.469972 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77f7b234-10af-422c-91a6-7486ba39fe22 None admin] RBAC: Adding query filter params (role.id=7c1fdac272a34be99661077b0f1b8014, scope.project.id=dd8011750cb34075a6f8106533f848c2, user.id=6c8f41499fe54de28740b35e4f20cf88) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:23.470283 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77f7b234-10af-422c-91a6-7486ba39fe22 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:23.471814 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77f7b234-10af-422c-91a6-7486ba39fe22 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:23.478828 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 136/273] 10.222.0.33 () {56 vars in 1374 bytes} [Sun Oct 9 00:42:23 2022] GET /identity/v3/role_assignments?user.id=6c8f41499fe54de28740b35e4f20cf88&scope.project.id=dd8011750cb34075a6f8106533f848c2&role.id=7c1fdac272a34be99661077b0f1b8014 => generated 608 bytes in 266 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:42:23.710209 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9800db2-86e1-40c4-847b-38e21d5163f3 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:23.781044 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:23.841667 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9800db2-86e1-40c4-847b-38e21d5163f3 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:23.852885 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77f7b234-10af-422c-91a6-7486ba39fe22 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:23.893451 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64929a58-9ac0-40b3-9a31-1d3c8c107da4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:23.895245 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64929a58-9ac0-40b3-9a31-1d3c8c107da4 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:23.895738 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64929a58-9ac0-40b3-9a31-1d3c8c107da4 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:23.896188 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64929a58-9ac0-40b3-9a31-1d3c8c107da4 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:23.910235 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64929a58-9ac0-40b3-9a31-1d3c8c107da4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:42:23.916464 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-64929a58-9ac0-40b3-9a31-1d3c8c107da4 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 09 00:42:23.916464 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:42:23.916464 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:42:23.916464 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:42:23.916464 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:42:23.916464 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:42:23.916464 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:42:23.916464 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:42:23.916464 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:42:23.916464 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:42:23.916464 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:42:23.916464 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:42:23.916464 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:42:23.916464 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:42:23.916464 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:42:23.916464 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:42:23.917301 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:42:23.917301 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:42:23.917301 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:42:23.917301 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:42:23.917301 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:42:23.917301 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:42:23.917301 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:42:23.917301 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:42:23.917301 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:42:23.917301 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:42:23.917301 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:42:23.917301 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:42:23.917301 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:42:23.917301 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:42:23.917301 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:42:23.917301 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:42:23.918065 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:42:23.918065 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:42:23.918065 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:42:23.918065 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:42:23.918065 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:42:23.918065 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:42:23.918065 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:42:23.918065 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:42:23.918065 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:42:23.918065 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:42:23.918065 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 09 00:42:23.918065 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:42:23.920927 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 138/274] 10.222.0.33 () {56 vars in 1099 bytes} [Sun Oct 9 00:42:23 2022] GET /identity/v3/roles/member => generated 84 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:42:23.928352 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64929a58-9ac0-40b3-9a31-1d3c8c107da4 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:23.945080 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:23.980395 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77f7b234-10af-422c-91a6-7486ba39fe22 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:24.020393 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4b04603-c981-4efb-a992-66e31b9e8cb1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:24.021033 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b04603-c981-4efb-a992-66e31b9e8cb1 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:24.021317 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b04603-c981-4efb-a992-66e31b9e8cb1 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:24.021592 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b04603-c981-4efb-a992-66e31b9e8cb1 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:24.022723 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4b04603-c981-4efb-a992-66e31b9e8cb1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:24.024251 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4b04603-c981-4efb-a992-66e31b9e8cb1 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:24.027378 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 137/275] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:42:23 2022] GET /identity/v3/domains/default => generated 199 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:42:24.032951 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4b04603-c981-4efb-a992-66e31b9e8cb1 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:24.067923 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64929a58-9ac0-40b3-9a31-1d3c8c107da4 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:24.088830 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:24.115081 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16f9fc54-a87b-4ccf-89b6-cd72ddfa9571 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:24.115764 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f9fc54-a87b-4ccf-89b6-cd72ddfa9571 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:24.116473 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f9fc54-a87b-4ccf-89b6-cd72ddfa9571 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:24.116745 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f9fc54-a87b-4ccf-89b6-cd72ddfa9571 None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:24.117411 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16f9fc54-a87b-4ccf-89b6-cd72ddfa9571 None admin] RBAC: Adding query filter params (name=member) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:24.117717 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16f9fc54-a87b-4ccf-89b6-cd72ddfa9571 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:24.120344 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16f9fc54-a87b-4ccf-89b6-cd72ddfa9571 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:24.131659 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 139/276] 10.222.0.33 () {56 vars in 1101 bytes} [Sun Oct 9 00:42:23 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:24.138585 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16f9fc54-a87b-4ccf-89b6-cd72ddfa9571 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:24.151004 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4b04603-c981-4efb-a992-66e31b9e8cb1 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:24.189503 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3121e6d3-02f0-45d5-8cd2-aca21cb55b84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:24.190481 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3121e6d3-02f0-45d5-8cd2-aca21cb55b84 None admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:24.190924 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3121e6d3-02f0-45d5-8cd2-aca21cb55b84 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:24.191201 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3121e6d3-02f0-45d5-8cd2-aca21cb55b84 None admin] PATH_INFO: `/v3/users` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:24.191893 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3121e6d3-02f0-45d5-8cd2-aca21cb55b84 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:24.193347 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3121e6d3-02f0-45d5-8cd2-aca21cb55b84 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:24.206687 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:24.227338 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 138/277] 10.222.0.33 () {60 vars in 1132 bytes} [Sun Oct 9 00:42:24 2022] POST /identity/v3/users => generated 293 bytes in 195 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 09 00:42:24.228824 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d222bcd8-f953-4adf-8dd6-96f777b73811 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:24.229119 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d222bcd8-f953-4adf-8dd6-96f777b73811 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:24.229403 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d222bcd8-f953-4adf-8dd6-96f777b73811 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:24.230785 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 139/278] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:42:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:42:24.236678 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-076a9578-e045-49e1-9ef3-3adcde42b48b None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:24.237002 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-076a9578-e045-49e1-9ef3-3adcde42b48b None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:24.237223 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-076a9578-e045-49e1-9ef3-3adcde42b48b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:24.262268 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-076a9578-e045-49e1-9ef3-3adcde42b48b None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:42:24.263518 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-076a9578-e045-49e1-9ef3-3adcde42b48b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:42:24.265767 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16f9fc54-a87b-4ccf-89b6-cd72ddfa9571 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:24.307216 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e844b15-0ac2-490f-947b-d744256848f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:24.308784 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e844b15-0ac2-490f-947b-d744256848f9 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:24.309207 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e844b15-0ac2-490f-947b-d744256848f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:24.309622 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e844b15-0ac2-490f-947b-d744256848f9 None admin] PATH_INFO: `/v3/users/69a924d093764214bd6a5df8180bc398` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:24.326675 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e844b15-0ac2-490f-947b-d744256848f9 None admin] RBAC: Authorizing `identity:get_user(user_id=69a924d093764214bd6a5df8180bc398)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:24.328325 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e844b15-0ac2-490f-947b-d744256848f9 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:24.330658 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 140/279] 10.222.0.33 () {56 vars in 1177 bytes} [Sun Oct 9 00:42:24 2022] GET /identity/v3/users/69a924d093764214bd6a5df8180bc398 => generated 293 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:24.337724 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e844b15-0ac2-490f-947b-d744256848f9 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:24.346562 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 140/280] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:42:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 09 00:42:24.377814 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:24.452470 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e844b15-0ac2-490f-947b-d744256848f9 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:24.487547 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5512647f-e954-486d-9654-59021892fe42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:24.488501 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5512647f-e954-486d-9654-59021892fe42 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:24.488970 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5512647f-e954-486d-9654-59021892fe42 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:24.489431 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5512647f-e954-486d-9654-59021892fe42 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:24.490337 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5512647f-e954-486d-9654-59021892fe42 None admin] RBAC: Adding query filter params (role.id=588016933bab436d8b7c3ac76827356d, user.id=69a924d093764214bd6a5df8180bc398, scope.system=all) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:24.490866 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5512647f-e954-486d-9654-59021892fe42 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:24.492635 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5512647f-e954-486d-9654-59021892fe42 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:24.497979 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 141/281] 10.222.0.33 () {56 vars in 1308 bytes} [Sun Oct 9 00:42:24 2022] GET /identity/v3/role_assignments?user.id=69a924d093764214bd6a5df8180bc398&scope.system=all&role.id=588016933bab436d8b7c3ac76827356d => generated 226 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:42:24.624342 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-076a9578-e045-49e1-9ef3-3adcde42b48b None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:24.737866 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-076a9578-e045-49e1-9ef3-3adcde42b48b None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:24.764453 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:24.776677 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da9e2321-881d-41b1-8b88-111350bdbb60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:24.777331 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da9e2321-881d-41b1-8b88-111350bdbb60 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:24.777612 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da9e2321-881d-41b1-8b88-111350bdbb60 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:24.777890 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da9e2321-881d-41b1-8b88-111350bdbb60 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:24.793945 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da9e2321-881d-41b1-8b88-111350bdbb60 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:42:24.798822 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-da9e2321-881d-41b1-8b88-111350bdbb60 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 09 00:42:24.798822 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:42:24.798822 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:42:24.798822 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:42:24.798822 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:42:24.798822 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:42:24.798822 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:42:24.798822 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:42:24.798822 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:42:24.798822 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:42:24.798822 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:42:24.798822 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:42:24.798822 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:42:24.798822 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:42:24.798822 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:42:24.798822 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:42:24.799685 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:42:24.799685 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:42:24.799685 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:42:24.799685 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:42:24.799685 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:42:24.799685 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:42:24.799685 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:42:24.799685 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:42:24.799685 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:42:24.799685 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:42:24.799685 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:42:24.799685 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:42:24.799685 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:42:24.799685 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:42:24.799685 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:42:24.799685 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:42:24.800455 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:42:24.800455 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:42:24.800455 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:42:24.800455 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:42:24.800455 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:42:24.800455 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:42:24.800455 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:42:24.800455 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:42:24.800455 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:42:24.800455 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:42:24.800455 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 09 00:42:24.800455 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:42:24.800455 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 141/282] 10.222.0.33 () {56 vars in 1099 bytes} [Sun Oct 9 00:42:24 2022] GET /identity/v3/roles/reader => generated 84 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:42:24.807856 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5512647f-e954-486d-9654-59021892fe42 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:24.913503 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:24.923442 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5512647f-e954-486d-9654-59021892fe42 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:24.974707 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6659376c-37c9-4934-88f4-a36302fbe601 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:24.975324 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6659376c-37c9-4934-88f4-a36302fbe601 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:24.975617 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6659376c-37c9-4934-88f4-a36302fbe601 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:24.975893 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6659376c-37c9-4934-88f4-a36302fbe601 None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:24.976558 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6659376c-37c9-4934-88f4-a36302fbe601 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:24.976853 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6659376c-37c9-4934-88f4-a36302fbe601 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:24.978797 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6659376c-37c9-4934-88f4-a36302fbe601 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:24.987811 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 142/283] 10.222.0.33 () {56 vars in 1101 bytes} [Sun Oct 9 00:42:24 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:24.994257 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da9e2321-881d-41b1-8b88-111350bdbb60 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:25.102101 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da9e2321-881d-41b1-8b88-111350bdbb60 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:25.142933 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29ae93ea-723e-46ea-92db-9af4a16ca313 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:25.143571 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ae93ea-723e-46ea-92db-9af4a16ca313 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:25.143878 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ae93ea-723e-46ea-92db-9af4a16ca313 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:25.144153 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ae93ea-723e-46ea-92db-9af4a16ca313 None admin] PATH_INFO: `/v3/users/13e948e5692e4a7ab9711a3062b73ad7` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:25.145296 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29ae93ea-723e-46ea-92db-9af4a16ca313 None admin] RBAC: Authorizing `identity:get_user(user_id=13e948e5692e4a7ab9711a3062b73ad7)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:25.146751 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29ae93ea-723e-46ea-92db-9af4a16ca313 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:25.148668 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 142/284] 10.222.0.33 () {56 vars in 1177 bytes} [Sun Oct 9 00:42:24 2022] GET /identity/v3/users/13e948e5692e4a7ab9711a3062b73ad7 => generated 297 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:25.154138 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6659376c-37c9-4934-88f4-a36302fbe601 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:25.308143 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6659376c-37c9-4934-88f4-a36302fbe601 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:25.358154 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4560d70d-bce4-47c7-b1b5-aef71ab6ebd1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:25.358792 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4560d70d-bce4-47c7-b1b5-aef71ab6ebd1 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:25.359071 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4560d70d-bce4-47c7-b1b5-aef71ab6ebd1 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:25.359347 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4560d70d-bce4-47c7-b1b5-aef71ab6ebd1 None admin] PATH_INFO: `/v3/projects/dd8011750cb34075a6f8106533f848c2` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:25.360503 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4560d70d-bce4-47c7-b1b5-aef71ab6ebd1 None admin] RBAC: Authorizing `identity:get_project(project_id=dd8011750cb34075a6f8106533f848c2)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:25.361936 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4560d70d-bce4-47c7-b1b5-aef71ab6ebd1 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:25.363869 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 143/285] 10.222.0.33 () {56 vars in 1186 bytes} [Sun Oct 9 00:42:25 2022] GET /identity/v3/projects/dd8011750cb34075a6f8106533f848c2 => generated 302 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:25.370060 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29ae93ea-723e-46ea-92db-9af4a16ca313 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:25.492502 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:25.543622 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29ae93ea-723e-46ea-92db-9af4a16ca313 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:25.612972 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d0f6524-88fd-4cc0-92a6-866e2e9924ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:25.613579 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0f6524-88fd-4cc0-92a6-866e2e9924ac None admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:25.613863 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0f6524-88fd-4cc0-92a6-866e2e9924ac None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:25.614141 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0f6524-88fd-4cc0-92a6-866e2e9924ac None admin] PATH_INFO: `/v3/projects/dd8011750cb34075a6f8106533f848c2/users/13e948e5692e4a7ab9711a3062b73ad7/roles/2d631b6eec1646c9834f74efa1d66091` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:25.616260 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d0f6524-88fd-4cc0-92a6-866e2e9924ac None admin] RBAC: Authorizing `identity:create_grant(project_id=dd8011750cb34075a6f8106533f848c2, user_id=13e948e5692e4a7ab9711a3062b73ad7, role_id=2d631b6eec1646c9834f74efa1d66091)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:25.617750 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d0f6524-88fd-4cc0-92a6-866e2e9924ac None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:25.645132 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 143/286] 10.222.0.33 () {58 vars in 1439 bytes} [Sun Oct 9 00:42:25 2022] PUT /identity/v3/projects/dd8011750cb34075a6f8106533f848c2/users/13e948e5692e4a7ab9711a3062b73ad7/roles/2d631b6eec1646c9834f74efa1d66091 => generated 0 bytes in 276 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:42:25.764700 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22c0e852-bee2-4ed8-b6e9-e1eb54c8fec6 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:25.765045 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22c0e852-bee2-4ed8-b6e9-e1eb54c8fec6 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:25.765247 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22c0e852-bee2-4ed8-b6e9-e1eb54c8fec6 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:25.766586 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 144/287] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:42:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:42:25.772356 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d4564de-c773-4c31-a40a-7e9e7d63e9de None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:25.772641 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d4564de-c773-4c31-a40a-7e9e7d63e9de None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:25.772911 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d4564de-c773-4c31-a40a-7e9e7d63e9de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:25.792482 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-5d4564de-c773-4c31-a40a-7e9e7d63e9de None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:42:25.793705 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-5d4564de-c773-4c31-a40a-7e9e7d63e9de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:42:25.867287 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 144/288] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:42:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 09 00:42:26.235530 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22c0e852-bee2-4ed8-b6e9-e1eb54c8fec6 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:26.269879 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:26.371854 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22c0e852-bee2-4ed8-b6e9-e1eb54c8fec6 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:26.414445 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad71af80-7fc9-433b-8073-08769008ab40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:26.415080 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad71af80-7fc9-433b-8073-08769008ab40 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:26.416006 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad71af80-7fc9-433b-8073-08769008ab40 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:26.416343 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad71af80-7fc9-433b-8073-08769008ab40 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:26.432574 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad71af80-7fc9-433b-8073-08769008ab40 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:42:26.437903 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-ad71af80-7fc9-433b-8073-08769008ab40 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 09 00:42:26.437903 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:42:26.437903 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:42:26.437903 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:42:26.437903 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:42:26.437903 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:42:26.437903 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:42:26.437903 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:42:26.437903 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:42:26.437903 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:42:26.437903 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:42:26.437903 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:42:26.437903 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:42:26.437903 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:42:26.437903 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:42:26.437903 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:42:26.438732 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:42:26.438732 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:42:26.438732 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:42:26.438732 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:42:26.438732 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:42:26.438732 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:42:26.438732 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:42:26.438732 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:42:26.438732 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:42:26.438732 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:42:26.438732 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:42:26.438732 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:42:26.438732 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:42:26.438732 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:42:26.438732 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:42:26.438732 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:42:26.439578 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:42:26.439578 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:42:26.439578 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:42:26.439578 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:42:26.439578 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:42:26.439578 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:42:26.439578 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:42:26.439578 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:42:26.439578 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:42:26.439578 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:42:26.439578 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 09 00:42:26.439578 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:42:26.439578 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 145/289] 10.222.0.33 () {56 vars in 1099 bytes} [Sun Oct 9 00:42:26 2022] GET /identity/v3/roles/member => generated 84 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:42:26.445183 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d4564de-c773-4c31-a40a-7e9e7d63e9de None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:26.512103 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:26.571409 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d4564de-c773-4c31-a40a-7e9e7d63e9de None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:26.615240 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61292e76-73ef-448c-af8a-da7dfa5f9051 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:26.619926 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61292e76-73ef-448c-af8a-da7dfa5f9051 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:26.620218 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61292e76-73ef-448c-af8a-da7dfa5f9051 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:26.620493 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61292e76-73ef-448c-af8a-da7dfa5f9051 None admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:26.621171 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61292e76-73ef-448c-af8a-da7dfa5f9051 None admin] RBAC: Adding query filter params (name=member) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:26.621474 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61292e76-73ef-448c-af8a-da7dfa5f9051 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:26.622849 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61292e76-73ef-448c-af8a-da7dfa5f9051 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:26.631856 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 145/290] 10.222.0.33 () {56 vars in 1101 bytes} [Sun Oct 9 00:42:26 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:26.637631 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad71af80-7fc9-433b-8073-08769008ab40 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:26.752197 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:26.778841 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad71af80-7fc9-433b-8073-08769008ab40 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:26.823551 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dd8507f-9296-4e0d-8a0a-99ec0f258799 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:26.824204 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dd8507f-9296-4e0d-8a0a-99ec0f258799 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:26.824486 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dd8507f-9296-4e0d-8a0a-99ec0f258799 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:26.824757 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dd8507f-9296-4e0d-8a0a-99ec0f258799 None admin] PATH_INFO: `/v3/users/65449b4b768e43ac9da8d8d112d2f701` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:26.825881 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dd8507f-9296-4e0d-8a0a-99ec0f258799 None admin] RBAC: Authorizing `identity:get_user(user_id=65449b4b768e43ac9da8d8d112d2f701)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:26.827327 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dd8507f-9296-4e0d-8a0a-99ec0f258799 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:26.829206 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 146/291] 10.222.0.33 () {56 vars in 1177 bytes} [Sun Oct 9 00:42:26 2022] GET /identity/v3/users/65449b4b768e43ac9da8d8d112d2f701 => generated 297 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:26.834533 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61292e76-73ef-448c-af8a-da7dfa5f9051 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:27.021272 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61292e76-73ef-448c-af8a-da7dfa5f9051 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:27.056722 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:27.087628 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06df0339-74e6-47dd-8003-46f208d38438 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:27.090378 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06df0339-74e6-47dd-8003-46f208d38438 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:27.090943 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06df0339-74e6-47dd-8003-46f208d38438 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:27.091216 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4052bde5-6ed3-466a-bc05-461cfe876630 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:27.091323 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06df0339-74e6-47dd-8003-46f208d38438 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:27.091505 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4052bde5-6ed3-466a-bc05-461cfe876630 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:27.091854 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4052bde5-6ed3-466a-bc05-461cfe876630 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:27.093422 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 147/292] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:42:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:42:27.098587 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7cf8573-61d1-406f-8784-f3a7826dd25c None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:27.098875 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7cf8573-61d1-406f-8784-f3a7826dd25c None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:27.099320 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7cf8573-61d1-406f-8784-f3a7826dd25c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:27.124194 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-c7cf8573-61d1-406f-8784-f3a7826dd25c None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:42:27.125384 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-c7cf8573-61d1-406f-8784-f3a7826dd25c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:42:27.204048 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 148/293] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:42:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 09 00:42:27.217236 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b15f5a-7b50-4c1a-8903-33ca935323c7 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:27.217511 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b15f5a-7b50-4c1a-8903-33ca935323c7 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:27.217787 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b15f5a-7b50-4c1a-8903-33ca935323c7 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:27.219281 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 149/294] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:42:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 09 00:42:27.225115 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e0819c-aded-4e26-8cf3-8337580e5f5b None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:27.225414 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e0819c-aded-4e26-8cf3-8337580e5f5b None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:27.225675 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e0819c-aded-4e26-8cf3-8337580e5f5b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:27.230713 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06df0339-74e6-47dd-8003-46f208d38438 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:27.232288 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06df0339-74e6-47dd-8003-46f208d38438 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:27.262900 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-02e0819c-aded-4e26-8cf3-8337580e5f5b None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:42:27.264128 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-02e0819c-aded-4e26-8cf3-8337580e5f5b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:42:27.385534 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 150/295] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:42:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 09 00:42:27.385534 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d2fb5c-6754-4522-aa4b-e96da738ac3f None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:27.385534 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d2fb5c-6754-4522-aa4b-e96da738ac3f None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:27.385534 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d2fb5c-6754-4522-aa4b-e96da738ac3f None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:27.385534 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 151/296] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:42:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:42:27.389671 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c85b170d-4545-4608-9276-264f036bb9ab None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:27.390122 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c85b170d-4545-4608-9276-264f036bb9ab None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:27.390409 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c85b170d-4545-4608-9276-264f036bb9ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:27.397727 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 146/297] 10.222.0.33 () {58 vars in 1282 bytes} [Sun Oct 9 00:42:26 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 564 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 09 00:42:27.408113 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06df0339-74e6-47dd-8003-46f208d38438 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:27.416938 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:27.436567 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-c85b170d-4545-4608-9276-264f036bb9ab None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:42:27.438060 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-c85b170d-4545-4608-9276-264f036bb9ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:42:27.455076 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:27.545171 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 152/298] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:42:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 09 00:42:27.546346 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c85b170d-4545-4608-9276-264f036bb9ab None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:27.563162 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06df0339-74e6-47dd-8003-46f208d38438 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:27.609957 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-945f4346-96d4-451b-b56c-377eca2dbec7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:27.610829 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-945f4346-96d4-451b-b56c-377eca2dbec7 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:27.611276 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-945f4346-96d4-451b-b56c-377eca2dbec7 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:27.611798 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-945f4346-96d4-451b-b56c-377eca2dbec7 None admin] PATH_INFO: `/v3/projects/dd8011750cb34075a6f8106533f848c2` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:27.613099 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-945f4346-96d4-451b-b56c-377eca2dbec7 None admin] RBAC: Authorizing `identity:get_project(project_id=dd8011750cb34075a6f8106533f848c2)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:27.614655 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-945f4346-96d4-451b-b56c-377eca2dbec7 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:27.616728 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 147/299] 10.222.0.33 () {56 vars in 1186 bytes} [Sun Oct 9 00:42:27 2022] GET /identity/v3/projects/dd8011750cb34075a6f8106533f848c2 => generated 302 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:27.627730 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-945f4346-96d4-451b-b56c-377eca2dbec7 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:27.709615 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c85b170d-4545-4608-9276-264f036bb9ab None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:27.741725 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:27.748938 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d2389a0-4d06-4255-90fd-deaecf47bc78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:27.749765 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2389a0-4d06-4255-90fd-deaecf47bc78 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:27.750217 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2389a0-4d06-4255-90fd-deaecf47bc78 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:27.750633 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2389a0-4d06-4255-90fd-deaecf47bc78 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:27.751932 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d2389a0-4d06-4255-90fd-deaecf47bc78 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:27.754382 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d2389a0-4d06-4255-90fd-deaecf47bc78 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:27.756194 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 153/300] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:42:27 2022] GET /identity/v3/domains/default => generated 199 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:27.757084 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d2389a0-4d06-4255-90fd-deaecf47bc78 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:27.773496 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:27.775199 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-945f4346-96d4-451b-b56c-377eca2dbec7 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:27.810669 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aae2437f-9f7a-440a-a4b9-a96e9e26ae18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:27.811493 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae2437f-9f7a-440a-a4b9-a96e9e26ae18 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:27.811934 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae2437f-9f7a-440a-a4b9-a96e9e26ae18 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:27.812377 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae2437f-9f7a-440a-a4b9-a96e9e26ae18 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:27.813232 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aae2437f-9f7a-440a-a4b9-a96e9e26ae18 None admin] RBAC: Adding query filter params (role.id=588016933bab436d8b7c3ac76827356d, scope.project.id=dd8011750cb34075a6f8106533f848c2, user.id=65449b4b768e43ac9da8d8d112d2f701) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:27.813675 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aae2437f-9f7a-440a-a4b9-a96e9e26ae18 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:27.815311 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aae2437f-9f7a-440a-a4b9-a96e9e26ae18 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:27.822284 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 148/301] 10.222.0.33 () {56 vars in 1374 bytes} [Sun Oct 9 00:42:27 2022] GET /identity/v3/role_assignments?user.id=65449b4b768e43ac9da8d8d112d2f701&scope.project.id=dd8011750cb34075a6f8106533f848c2&role.id=588016933bab436d8b7c3ac76827356d => generated 608 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:42:27.827623 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aae2437f-9f7a-440a-a4b9-a96e9e26ae18 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:27.894966 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d2389a0-4d06-4255-90fd-deaecf47bc78 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:27.936527 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfd060da-91cb-49d3-92e3-b43df39f035e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:27.937588 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfd060da-91cb-49d3-92e3-b43df39f035e None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:27.938226 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfd060da-91cb-49d3-92e3-b43df39f035e None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:27.938782 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfd060da-91cb-49d3-92e3-b43df39f035e None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:27.941684 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aae2437f-9f7a-440a-a4b9-a96e9e26ae18 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:27.952275 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfd060da-91cb-49d3-92e3-b43df39f035e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:42:27.958073 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-bfd060da-91cb-49d3-92e3-b43df39f035e None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 09 00:42:27.958073 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:42:27.958073 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:42:27.958073 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:42:27.958073 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:42:27.958073 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:42:27.958073 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:42:27.958073 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:42:27.958073 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:42:27.958073 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:42:27.958073 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:42:27.958073 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:42:27.958073 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:42:27.958073 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:42:27.958073 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:42:27.958073 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:42:27.958983 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:42:27.958983 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:42:27.958983 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:42:27.958983 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:42:27.958983 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:42:27.958983 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:42:27.958983 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:42:27.958983 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:42:27.958983 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:42:27.958983 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:42:27.958983 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:42:27.958983 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:42:27.958983 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:42:27.958983 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:42:27.958983 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:42:27.958983 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:42:27.960236 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:42:27.960236 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:42:27.960236 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:42:27.960236 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:42:27.960236 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:42:27.960236 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:42:27.960236 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:42:27.960236 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:42:27.960236 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:42:27.960236 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:42:27.960236 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 09 00:42:27.960236 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:42:27.962883 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 154/302] 10.222.0.33 () {56 vars in 1099 bytes} [Sun Oct 9 00:42:27 2022] GET /identity/v3/roles/reader => generated 84 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:42:27.964134 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfd060da-91cb-49d3-92e3-b43df39f035e None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:27.979084 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa19483d-fc0d-43f7-9c09-857a8b7ea07b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:27.979813 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa19483d-fc0d-43f7-9c09-857a8b7ea07b None admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:27.980095 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa19483d-fc0d-43f7-9c09-857a8b7ea07b None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:27.980419 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa19483d-fc0d-43f7-9c09-857a8b7ea07b None admin] PATH_INFO: `/v3/groups` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:27.981152 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa19483d-fc0d-43f7-9c09-857a8b7ea07b None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:27.982650 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa19483d-fc0d-43f7-9c09-857a8b7ea07b None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:27.996052 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:28.003239 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 149/303] 10.222.0.33 () {60 vars in 1134 bytes} [Sun Oct 9 00:42:27 2022] POST /identity/v3/groups => generated 229 bytes in 180 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:42:28.009036 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa19483d-fc0d-43f7-9c09-857a8b7ea07b None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:28.062405 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:28.130103 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfd060da-91cb-49d3-92e3-b43df39f035e None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:28.169752 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa19483d-fc0d-43f7-9c09-857a8b7ea07b None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:28.182790 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-592e91ae-1d93-4207-abba-f1bb7f798306 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:28.183842 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-592e91ae-1d93-4207-abba-f1bb7f798306 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:28.184339 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-592e91ae-1d93-4207-abba-f1bb7f798306 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:28.184804 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-592e91ae-1d93-4207-abba-f1bb7f798306 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:28.198164 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-592e91ae-1d93-4207-abba-f1bb7f798306 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:42:28.203443 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-592e91ae-1d93-4207-abba-f1bb7f798306 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 09 00:42:28.203443 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:42:28.203443 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:42:28.203443 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:42:28.203443 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:42:28.203443 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:42:28.203443 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:42:28.203443 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:42:28.203443 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:42:28.203443 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:42:28.203443 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:42:28.203443 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:42:28.203443 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:42:28.203443 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:42:28.203443 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:42:28.203443 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:42:28.204495 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:42:28.204495 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:42:28.204495 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:42:28.204495 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:42:28.204495 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:42:28.204495 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:42:28.204495 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:42:28.204495 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:42:28.204495 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:42:28.204495 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:42:28.204495 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:42:28.204495 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:42:28.204495 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:42:28.204495 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:42:28.204495 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:42:28.204495 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:42:28.205399 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:42:28.205399 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:42:28.205399 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:42:28.205399 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:42:28.205399 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:42:28.205399 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:42:28.205399 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:42:28.205399 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:42:28.205399 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:42:28.205399 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:42:28.205399 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 09 00:42:28.205399 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:42:28.205399 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 155/304] 10.222.0.33 () {56 vars in 1099 bytes} [Sun Oct 9 00:42:27 2022] GET /identity/v3/roles/member => generated 84 bytes in 241 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:42:28.206039 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe8f38c-db3a-45f9-a064-26d11a56d33a None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:28.206103 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe8f38c-db3a-45f9-a064-26d11a56d33a None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:28.206683 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe8f38c-db3a-45f9-a064-26d11a56d33a None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:28.207812 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 156/305] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:42:28 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:42:28.213057 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fe8f38c-db3a-45f9-a064-26d11a56d33a None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:28.215688 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92968d25-d43d-4786-8778-eeaff2b6a7da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:28.216321 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92968d25-d43d-4786-8778-eeaff2b6a7da None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:28.216600 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92968d25-d43d-4786-8778-eeaff2b6a7da None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:28.216872 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92968d25-d43d-4786-8778-eeaff2b6a7da None admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:28.217781 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92968d25-d43d-4786-8778-eeaff2b6a7da None admin] RBAC: Adding query filter params (name=reader) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:28.218089 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92968d25-d43d-4786-8778-eeaff2b6a7da None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:28.220458 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92968d25-d43d-4786-8778-eeaff2b6a7da None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:28.232367 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 150/306] 10.222.0.33 () {56 vars in 1101 bytes} [Sun Oct 9 00:42:28 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:28.233472 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-119e5350-762f-4886-ab05-074df20b8c2d None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:28.233755 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-119e5350-762f-4886-ab05-074df20b8c2d None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:28.234029 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-119e5350-762f-4886-ab05-074df20b8c2d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:28.254899 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-119e5350-762f-4886-ab05-074df20b8c2d None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:42:28.256133 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-119e5350-762f-4886-ab05-074df20b8c2d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:42:28.322173 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:28.335177 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 151/307] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:42:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 09 00:42:28.336118 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-119e5350-762f-4886-ab05-074df20b8c2d None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:28.355619 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fe8f38c-db3a-45f9-a064-26d11a56d33a None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:28.393014 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:28.398638 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4b039db-77cb-474a-a99b-f6822db0f3b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:28.399245 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b039db-77cb-474a-a99b-f6822db0f3b2 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:28.399644 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b039db-77cb-474a-a99b-f6822db0f3b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:28.399968 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b039db-77cb-474a-a99b-f6822db0f3b2 None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:28.401271 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4b039db-77cb-474a-a99b-f6822db0f3b2 None admin] RBAC: Adding query filter params (name=member) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:28.401929 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4b039db-77cb-474a-a99b-f6822db0f3b2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:28.404328 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4b039db-77cb-474a-a99b-f6822db0f3b2 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:28.421534 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 157/308] 10.222.0.33 () {56 vars in 1101 bytes} [Sun Oct 9 00:42:28 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:28.422534 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4b039db-77cb-474a-a99b-f6822db0f3b2 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:28.455957 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-119e5350-762f-4886-ab05-074df20b8c2d None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:28.493119 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-135c58aa-33cf-406b-8ac8-7e1886565ee9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:28.494145 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-135c58aa-33cf-406b-8ac8-7e1886565ee9 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:28.494713 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-135c58aa-33cf-406b-8ac8-7e1886565ee9 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:28.495180 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-135c58aa-33cf-406b-8ac8-7e1886565ee9 None admin] PATH_INFO: `/v3/users/ea9a35b7be1c4625a0f385ee980e96ea` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:28.510689 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-135c58aa-33cf-406b-8ac8-7e1886565ee9 None admin] RBAC: Authorizing `identity:get_user(user_id=ea9a35b7be1c4625a0f385ee980e96ea)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:28.512418 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-135c58aa-33cf-406b-8ac8-7e1886565ee9 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:28.514565 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 152/309] 10.222.0.33 () {56 vars in 1177 bytes} [Sun Oct 9 00:42:28 2022] GET /identity/v3/users/ea9a35b7be1c4625a0f385ee980e96ea => generated 293 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:28.525255 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-135c58aa-33cf-406b-8ac8-7e1886565ee9 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:28.543543 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4b039db-77cb-474a-a99b-f6822db0f3b2 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:28.581416 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:28.598569 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbb86142-f597-4197-8932-18318d549869 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:28.599693 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb86142-f597-4197-8932-18318d549869 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:28.600044 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb86142-f597-4197-8932-18318d549869 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:28.600346 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb86142-f597-4197-8932-18318d549869 None admin] PATH_INFO: `/v3/users/69a924d093764214bd6a5df8180bc398` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:28.601654 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbb86142-f597-4197-8932-18318d549869 None admin] RBAC: Authorizing `identity:get_user(user_id=69a924d093764214bd6a5df8180bc398)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:28.607614 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbb86142-f597-4197-8932-18318d549869 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:28.607614 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 158/310] 10.222.0.33 () {56 vars in 1177 bytes} [Sun Oct 9 00:42:28 2022] GET /identity/v3/users/69a924d093764214bd6a5df8180bc398 => generated 293 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:28.608704 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbb86142-f597-4197-8932-18318d549869 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:28.646013 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-135c58aa-33cf-406b-8ac8-7e1886565ee9 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:28.675002 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:28.682250 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e5a220a-03a0-4ce3-bf22-fbe73c686118 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:28.682847 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e5a220a-03a0-4ce3-bf22-fbe73c686118 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:28.683121 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e5a220a-03a0-4ce3-bf22-fbe73c686118 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:28.683391 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e5a220a-03a0-4ce3-bf22-fbe73c686118 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:28.684138 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e5a220a-03a0-4ce3-bf22-fbe73c686118 None admin] RBAC: Adding query filter params (role.id=2d631b6eec1646c9834f74efa1d66091, user.id=ea9a35b7be1c4625a0f385ee980e96ea, scope.system=all) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:28.684441 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e5a220a-03a0-4ce3-bf22-fbe73c686118 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:28.685859 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e5a220a-03a0-4ce3-bf22-fbe73c686118 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:28.695636 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 153/311] 10.222.0.33 () {56 vars in 1308 bytes} [Sun Oct 9 00:42:28 2022] GET /identity/v3/role_assignments?user.id=ea9a35b7be1c4625a0f385ee980e96ea&scope.system=all&role.id=2d631b6eec1646c9834f74efa1d66091 => generated 226 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:28.695636 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e5a220a-03a0-4ce3-bf22-fbe73c686118 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:28.725182 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbb86142-f597-4197-8932-18318d549869 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:28.765710 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a228a8f5-1713-48f3-859d-7169c293159b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:28.766433 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a228a8f5-1713-48f3-859d-7169c293159b None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:28.766810 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a228a8f5-1713-48f3-859d-7169c293159b None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:28.767158 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a228a8f5-1713-48f3-859d-7169c293159b None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:28.778978 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a228a8f5-1713-48f3-859d-7169c293159b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:42:28.785343 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-a228a8f5-1713-48f3-859d-7169c293159b None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 09 00:42:28.785343 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:42:28.785343 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:42:28.785343 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:42:28.785343 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:42:28.785343 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:42:28.785343 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:42:28.785343 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:42:28.785343 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:42:28.785343 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:42:28.785343 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:42:28.785343 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:42:28.785343 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:42:28.785343 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:42:28.785343 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:42:28.785343 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:42:28.786262 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:42:28.786262 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:42:28.786262 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:42:28.786262 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:42:28.786262 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:42:28.786262 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:42:28.786262 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:42:28.786262 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:42:28.786262 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:42:28.786262 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:42:28.786262 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:42:28.786262 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:42:28.786262 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:42:28.786262 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:42:28.786262 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:42:28.786262 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:42:28.787026 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:42:28.787026 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:42:28.787026 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:42:28.787026 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:42:28.787026 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:42:28.787026 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:42:28.787026 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:42:28.787026 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:42:28.787026 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:42:28.787026 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:42:28.787026 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 09 00:42:28.787026 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:42:28.791873 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 159/312] 10.222.0.33 () {56 vars in 1099 bytes} [Sun Oct 9 00:42:28 2022] GET /identity/v3/roles/reader => generated 84 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:42:28.792967 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a228a8f5-1713-48f3-859d-7169c293159b None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:28.841425 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e5a220a-03a0-4ce3-bf22-fbe73c686118 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:28.860459 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:28.887115 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c079fe2c-e06b-48c8-a9d0-19df432a00e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:28.888016 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c079fe2c-e06b-48c8-a9d0-19df432a00e0 None admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:28.888397 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c079fe2c-e06b-48c8-a9d0-19df432a00e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:28.888790 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c079fe2c-e06b-48c8-a9d0-19df432a00e0 None admin] PATH_INFO: `/v3/system/users/69a924d093764214bd6a5df8180bc398/roles/588016933bab436d8b7c3ac76827356d` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:28.890609 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c079fe2c-e06b-48c8-a9d0-19df432a00e0 None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=69a924d093764214bd6a5df8180bc398, role_id=588016933bab436d8b7c3ac76827356d)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:28.891883 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c079fe2c-e06b-48c8-a9d0-19df432a00e0 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:28.904331 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 154/313] 10.222.0.33 () {58 vars in 1334 bytes} [Sun Oct 9 00:42:28 2022] PUT /identity/v3/system/users/69a924d093764214bd6a5df8180bc398/roles/588016933bab436d8b7c3ac76827356d => generated 0 bytes in 213 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:42:28.920505 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a228a8f5-1713-48f3-859d-7169c293159b None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:28.952428 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:28.966758 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35ad5ad8-299b-4d72-bbc8-70f3178df865 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:28.967418 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ad5ad8-299b-4d72-bbc8-70f3178df865 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:28.967777 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ad5ad8-299b-4d72-bbc8-70f3178df865 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:28.968053 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ad5ad8-299b-4d72-bbc8-70f3178df865 None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:28.968740 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35ad5ad8-299b-4d72-bbc8-70f3178df865 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:28.969046 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35ad5ad8-299b-4d72-bbc8-70f3178df865 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:28.970490 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35ad5ad8-299b-4d72-bbc8-70f3178df865 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:28.980684 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 160/314] 10.222.0.33 () {56 vars in 1101 bytes} [Sun Oct 9 00:42:28 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:28.986585 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c079fe2c-e06b-48c8-a9d0-19df432a00e0 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:29.106976 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c079fe2c-e06b-48c8-a9d0-19df432a00e0 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:29.145361 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-555097fb-2d51-4de0-9500-11f694a1d930 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:29.146041 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555097fb-2d51-4de0-9500-11f694a1d930 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:29.146373 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555097fb-2d51-4de0-9500-11f694a1d930 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:29.146659 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555097fb-2d51-4de0-9500-11f694a1d930 None admin] PATH_INFO: `/v3/users/13e948e5692e4a7ab9711a3062b73ad7` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:29.147837 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-555097fb-2d51-4de0-9500-11f694a1d930 None admin] RBAC: Authorizing `identity:get_user(user_id=13e948e5692e4a7ab9711a3062b73ad7)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:29.149348 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-555097fb-2d51-4de0-9500-11f694a1d930 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:29.151277 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 155/315] 10.222.0.33 () {56 vars in 1177 bytes} [Sun Oct 9 00:42:28 2022] GET /identity/v3/users/13e948e5692e4a7ab9711a3062b73ad7 => generated 297 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:29.156646 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35ad5ad8-299b-4d72-bbc8-70f3178df865 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:29.263452 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35ad5ad8-299b-4d72-bbc8-70f3178df865 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:29.303476 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9969338c-bc00-4965-afa6-c3c929465176 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:29.304111 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9969338c-bc00-4965-afa6-c3c929465176 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:29.304387 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9969338c-bc00-4965-afa6-c3c929465176 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:29.304661 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9969338c-bc00-4965-afa6-c3c929465176 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:29.359364 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:29.376737 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9969338c-bc00-4965-afa6-c3c929465176 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:29.378192 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9969338c-bc00-4965-afa6-c3c929465176 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:29.486442 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 161/316] 10.222.0.33 () {58 vars in 1282 bytes} [Sun Oct 9 00:42:29 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 330 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 09 00:42:29.492679 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-555097fb-2d51-4de0-9500-11f694a1d930 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:29.543438 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:29.599712 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-555097fb-2d51-4de0-9500-11f694a1d930 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:29.637677 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32c5bf9f-493c-40ae-a6fe-af1b344db9bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:29.638297 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c5bf9f-493c-40ae-a6fe-af1b344db9bd None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:29.638585 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c5bf9f-493c-40ae-a6fe-af1b344db9bd None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:29.638864 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c5bf9f-493c-40ae-a6fe-af1b344db9bd None admin] PATH_INFO: `/v3/projects/dd8011750cb34075a6f8106533f848c2` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:29.640039 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32c5bf9f-493c-40ae-a6fe-af1b344db9bd None admin] RBAC: Authorizing `identity:get_project(project_id=dd8011750cb34075a6f8106533f848c2)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:29.641538 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32c5bf9f-493c-40ae-a6fe-af1b344db9bd None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:29.643524 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 156/317] 10.222.0.33 () {56 vars in 1186 bytes} [Sun Oct 9 00:42:29 2022] GET /identity/v3/projects/dd8011750cb34075a6f8106533f848c2 => generated 302 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:29.649429 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9969338c-bc00-4965-afa6-c3c929465176 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:29.739038 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:29.761547 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9969338c-bc00-4965-afa6-c3c929465176 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:29.797843 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba31e7b2-f224-437c-a300-985eec5a78f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:29.798757 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba31e7b2-f224-437c-a300-985eec5a78f8 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:29.799277 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba31e7b2-f224-437c-a300-985eec5a78f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:29.799778 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba31e7b2-f224-437c-a300-985eec5a78f8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:29.800696 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba31e7b2-f224-437c-a300-985eec5a78f8 None admin] RBAC: Adding query filter params (role.id=2d631b6eec1646c9834f74efa1d66091, scope.project.id=dd8011750cb34075a6f8106533f848c2, user.id=13e948e5692e4a7ab9711a3062b73ad7) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:29.801201 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba31e7b2-f224-437c-a300-985eec5a78f8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:29.802819 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba31e7b2-f224-437c-a300-985eec5a78f8 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:29.809985 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 162/318] 10.222.0.33 () {56 vars in 1374 bytes} [Sun Oct 9 00:42:29 2022] GET /identity/v3/role_assignments?user.id=13e948e5692e4a7ab9711a3062b73ad7&scope.project.id=dd8011750cb34075a6f8106533f848c2&role.id=2d631b6eec1646c9834f74efa1d66091 => generated 608 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:30.390358 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e1001c-6c74-4f5a-aa83-922aad736985 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:30.390845 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e1001c-6c74-4f5a-aa83-922aad736985 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:30.390845 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e1001c-6c74-4f5a-aa83-922aad736985 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:30.392179 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 157/319] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:42:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:42:30.397496 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c118bf4-af69-4ca2-8995-48e62582c1ee None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:30.397851 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c118bf4-af69-4ca2-8995-48e62582c1ee None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:30.398192 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c118bf4-af69-4ca2-8995-48e62582c1ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:30.417122 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-3c118bf4-af69-4ca2-8995-48e62582c1ee None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:42:30.418274 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-3c118bf4-af69-4ca2-8995-48e62582c1ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:42:30.486093 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 163/320] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:42:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 09 00:42:30.749832 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46e1001c-6c74-4f5a-aa83-922aad736985 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:30.858766 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46e1001c-6c74-4f5a-aa83-922aad736985 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:30.872615 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:30.895123 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bae67927-e2a3-4ae9-bed1-27c41e9ae5c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:30.895785 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bae67927-e2a3-4ae9-bed1-27c41e9ae5c4 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:30.896066 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bae67927-e2a3-4ae9-bed1-27c41e9ae5c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:30.896392 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bae67927-e2a3-4ae9-bed1-27c41e9ae5c4 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:30.897484 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bae67927-e2a3-4ae9-bed1-27c41e9ae5c4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:30.898909 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bae67927-e2a3-4ae9-bed1-27c41e9ae5c4 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:30.903735 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 158/321] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:42:30 2022] GET /identity/v3/domains/default => generated 199 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:30.905999 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c118bf4-af69-4ca2-8995-48e62582c1ee None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:30.910888 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b729ed09-134a-4e82-895c-0399e17f8f84 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:30.911172 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b729ed09-134a-4e82-895c-0399e17f8f84 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:30.911448 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b729ed09-134a-4e82-895c-0399e17f8f84 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:30.912874 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 159/322] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:42:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 09 00:42:30.917811 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19e7e429-7fa7-4f59-84d3-d4e426715e50 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:30.918092 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19e7e429-7fa7-4f59-84d3-d4e426715e50 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:30.918367 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19e7e429-7fa7-4f59-84d3-d4e426715e50 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:30.936899 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-19e7e429-7fa7-4f59-84d3-d4e426715e50 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:42:30.938393 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-19e7e429-7fa7-4f59-84d3-d4e426715e50 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:42:30.989816 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:31.011119 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 160/323] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:42:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 09 00:42:31.013725 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c118bf4-af69-4ca2-8995-48e62582c1ee None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:31.052293 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2bd5444-4e2f-46c5-8b01-82ed83a8af70 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:31.052910 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2bd5444-4e2f-46c5-8b01-82ed83a8af70 None admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:31.053187 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2bd5444-4e2f-46c5-8b01-82ed83a8af70 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:31.053457 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2bd5444-4e2f-46c5-8b01-82ed83a8af70 None admin] PATH_INFO: `/v3/groups` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:31.054086 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2bd5444-4e2f-46c5-8b01-82ed83a8af70 None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:31.055558 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2bd5444-4e2f-46c5-8b01-82ed83a8af70 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:31.088016 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 164/324] 10.222.0.33 () {60 vars in 1134 bytes} [Sun Oct 9 00:42:30 2022] POST /identity/v3/groups => generated 226 bytes in 182 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:42:31.143652 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9292d95b-3760-4157-b6fb-8b516ff08a09 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:31.143960 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9292d95b-3760-4157-b6fb-8b516ff08a09 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:31.144216 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9292d95b-3760-4157-b6fb-8b516ff08a09 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:31.145759 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 161/325] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:42:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:42:31.151967 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5188120-98ba-42b4-a02b-0cbe85bd569b None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:31.152256 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5188120-98ba-42b4-a02b-0cbe85bd569b None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:31.152521 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5188120-98ba-42b4-a02b-0cbe85bd569b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:31.172910 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-a5188120-98ba-42b4-a02b-0cbe85bd569b None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:42:31.174206 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-a5188120-98ba-42b4-a02b-0cbe85bd569b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:42:31.249233 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 165/326] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:42:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 09 00:42:31.286197 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9292d95b-3760-4157-b6fb-8b516ff08a09 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:31.396455 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9292d95b-3760-4157-b6fb-8b516ff08a09 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:31.408523 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:31.433618 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff6b0b99-2478-4a41-bed8-f848b658b024 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:31.434304 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6b0b99-2478-4a41-bed8-f848b658b024 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:31.434585 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6b0b99-2478-4a41-bed8-f848b658b024 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:31.434855 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6b0b99-2478-4a41-bed8-f848b658b024 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:31.445918 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff6b0b99-2478-4a41-bed8-f848b658b024 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:42:31.451492 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-ff6b0b99-2478-4a41-bed8-f848b658b024 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 09 00:42:31.451492 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:42:31.451492 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:42:31.451492 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:42:31.451492 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:42:31.451492 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:42:31.451492 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:42:31.451492 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:42:31.451492 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:42:31.451492 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:42:31.451492 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:42:31.451492 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:42:31.451492 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:42:31.451492 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:42:31.451492 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:42:31.451492 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:42:31.452484 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:42:31.452484 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:42:31.452484 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:42:31.452484 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:42:31.452484 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:42:31.452484 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:42:31.452484 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:42:31.452484 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:42:31.452484 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:42:31.452484 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:42:31.452484 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:42:31.452484 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:42:31.452484 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:42:31.452484 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:42:31.452484 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:42:31.452484 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:42:31.453248 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:42:31.453248 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:42:31.453248 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:42:31.453248 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:42:31.453248 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:42:31.453248 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:42:31.453248 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:42:31.453248 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:42:31.453248 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:42:31.453248 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:42:31.453248 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 09 00:42:31.453248 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:42:31.453248 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 162/327] 10.222.0.33 () {56 vars in 1099 bytes} [Sun Oct 9 00:42:31 2022] GET /identity/v3/roles/reader => generated 84 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:42:31.458507 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5188120-98ba-42b4-a02b-0cbe85bd569b None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:31.522819 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff6b0b99-2478-4a41-bed8-f848b658b024 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:31.542194 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:31.581524 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5188120-98ba-42b4-a02b-0cbe85bd569b None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:31.620864 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc04b20a-c9da-4753-af0f-32639c7b3755 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:31.622100 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc04b20a-c9da-4753-af0f-32639c7b3755 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:31.622380 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc04b20a-c9da-4753-af0f-32639c7b3755 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:31.622651 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc04b20a-c9da-4753-af0f-32639c7b3755 None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:31.623325 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc04b20a-c9da-4753-af0f-32639c7b3755 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:31.623637 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc04b20a-c9da-4753-af0f-32639c7b3755 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:31.625436 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc04b20a-c9da-4753-af0f-32639c7b3755 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:31.645428 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 166/328] 10.222.0.33 () {56 vars in 1101 bytes} [Sun Oct 9 00:42:31 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:31.646698 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc04b20a-c9da-4753-af0f-32639c7b3755 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:31.664591 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff6b0b99-2478-4a41-bed8-f848b658b024 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:31.714750 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36abc8ae-1fba-4245-b2bd-5799c84311c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:31.715388 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36abc8ae-1fba-4245-b2bd-5799c84311c5 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:31.715708 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36abc8ae-1fba-4245-b2bd-5799c84311c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:31.716086 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36abc8ae-1fba-4245-b2bd-5799c84311c5 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:31.727690 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36abc8ae-1fba-4245-b2bd-5799c84311c5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:42:31.735069 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-36abc8ae-1fba-4245-b2bd-5799c84311c5 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 09 00:42:31.735069 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:42:31.735069 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:42:31.735069 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:42:31.735069 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:42:31.735069 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:42:31.735069 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:42:31.735069 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:42:31.735069 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:42:31.735069 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:42:31.735069 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:42:31.735069 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:42:31.735069 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:42:31.735069 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:42:31.735069 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:42:31.735069 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:42:31.735999 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:42:31.735999 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:42:31.735999 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:42:31.735999 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:42:31.735999 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:42:31.735999 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:42:31.735999 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:42:31.735999 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:42:31.735999 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:42:31.735999 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:42:31.735999 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:42:31.735999 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:42:31.735999 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:42:31.735999 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:42:31.735999 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:42:31.735999 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:42:31.736831 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:42:31.736831 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:42:31.736831 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:42:31.736831 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:42:31.736831 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:42:31.736831 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:42:31.736831 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:42:31.736831 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:42:31.736831 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:42:31.736831 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:42:31.736831 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 09 00:42:31.736831 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:42:31.736831 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 163/329] 10.222.0.33 () {56 vars in 1099 bytes} [Sun Oct 9 00:42:31 2022] GET /identity/v3/roles/member => generated 84 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:42:31.744764 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36abc8ae-1fba-4245-b2bd-5799c84311c5 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:31.780930 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:31.781140 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc04b20a-c9da-4753-af0f-32639c7b3755 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:31.812410 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:31.819014 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e531cd97-9d07-4d2d-b85b-4d818b7ac1bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:31.819935 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e531cd97-9d07-4d2d-b85b-4d818b7ac1bf None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:31.820383 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e531cd97-9d07-4d2d-b85b-4d818b7ac1bf None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:31.820875 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e531cd97-9d07-4d2d-b85b-4d818b7ac1bf None admin] PATH_INFO: `/v3/users/ea9a35b7be1c4625a0f385ee980e96ea` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:31.822549 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e531cd97-9d07-4d2d-b85b-4d818b7ac1bf None admin] RBAC: Authorizing `identity:get_user(user_id=ea9a35b7be1c4625a0f385ee980e96ea)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:31.824042 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e531cd97-9d07-4d2d-b85b-4d818b7ac1bf None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:31.825614 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 167/330] 10.222.0.33 () {56 vars in 1177 bytes} [Sun Oct 9 00:42:31 2022] GET /identity/v3/users/ea9a35b7be1c4625a0f385ee980e96ea => generated 293 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:42:31.831994 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e531cd97-9d07-4d2d-b85b-4d818b7ac1bf None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:31.870994 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36abc8ae-1fba-4245-b2bd-5799c84311c5 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:31.912762 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51d90def-fe57-4d23-8904-56c6da2bc866 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:31.913632 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d90def-fe57-4d23-8904-56c6da2bc866 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:31.913911 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d90def-fe57-4d23-8904-56c6da2bc866 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:31.914184 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d90def-fe57-4d23-8904-56c6da2bc866 None admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:31.916547 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51d90def-fe57-4d23-8904-56c6da2bc866 None admin] RBAC: Adding query filter params (name=member) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:31.916882 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51d90def-fe57-4d23-8904-56c6da2bc866 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:31.918269 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51d90def-fe57-4d23-8904-56c6da2bc866 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:31.928816 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 164/331] 10.222.0.33 () {56 vars in 1101 bytes} [Sun Oct 9 00:42:31 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:42:31.934697 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51d90def-fe57-4d23-8904-56c6da2bc866 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:31.954540 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e531cd97-9d07-4d2d-b85b-4d818b7ac1bf None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:31.995512 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35f2a27e-1b44-4889-8506-55c6f73ef1c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:31.996211 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f2a27e-1b44-4889-8506-55c6f73ef1c2 None admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:31.996489 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f2a27e-1b44-4889-8506-55c6f73ef1c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:31.996766 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f2a27e-1b44-4889-8506-55c6f73ef1c2 None admin] PATH_INFO: `/v3/system/users/ea9a35b7be1c4625a0f385ee980e96ea/roles/2d631b6eec1646c9834f74efa1d66091` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:31.998188 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35f2a27e-1b44-4889-8506-55c6f73ef1c2 None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=ea9a35b7be1c4625a0f385ee980e96ea, role_id=2d631b6eec1646c9834f74efa1d66091)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:31.999565 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35f2a27e-1b44-4889-8506-55c6f73ef1c2 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:32.014533 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 168/332] 10.222.0.33 () {58 vars in 1334 bytes} [Sun Oct 9 00:42:31 2022] PUT /identity/v3/system/users/ea9a35b7be1c4625a0f385ee980e96ea/roles/2d631b6eec1646c9834f74efa1d66091 => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 09 00:42:32.055876 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:32.065320 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51d90def-fe57-4d23-8904-56c6da2bc866 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:32.104702 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfbffc50-c71c-4c15-ad7b-4a51fad0b9da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:32.105348 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfbffc50-c71c-4c15-ad7b-4a51fad0b9da None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:32.105626 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfbffc50-c71c-4c15-ad7b-4a51fad0b9da None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:32.105904 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfbffc50-c71c-4c15-ad7b-4a51fad0b9da None admin] PATH_INFO: `/v3/users/69a924d093764214bd6a5df8180bc398` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:32.107085 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfbffc50-c71c-4c15-ad7b-4a51fad0b9da None admin] RBAC: Authorizing `identity:get_user(user_id=69a924d093764214bd6a5df8180bc398)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:32.108578 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfbffc50-c71c-4c15-ad7b-4a51fad0b9da None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:32.110587 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 165/333] 10.222.0.33 () {56 vars in 1177 bytes} [Sun Oct 9 00:42:31 2022] GET /identity/v3/users/69a924d093764214bd6a5df8180bc398 => generated 293 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:42:32.116257 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35f2a27e-1b44-4889-8506-55c6f73ef1c2 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:32.190302 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:32.245546 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35f2a27e-1b44-4889-8506-55c6f73ef1c2 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:32.285988 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e3bda39-d97e-4bfa-8515-77633b109e20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:32.286601 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e3bda39-d97e-4bfa-8515-77633b109e20 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:32.286881 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e3bda39-d97e-4bfa-8515-77633b109e20 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:32.287155 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e3bda39-d97e-4bfa-8515-77633b109e20 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:32.287906 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e3bda39-d97e-4bfa-8515-77633b109e20 None admin] RBAC: Adding query filter params (role.id=588016933bab436d8b7c3ac76827356d, user.id=69a924d093764214bd6a5df8180bc398, scope.system=all) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:32.288210 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e3bda39-d97e-4bfa-8515-77633b109e20 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:32.289961 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e3bda39-d97e-4bfa-8515-77633b109e20 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:32.304266 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 169/334] 10.222.0.33 () {56 vars in 1308 bytes} [Sun Oct 9 00:42:32 2022] GET /identity/v3/role_assignments?user.id=69a924d093764214bd6a5df8180bc398&scope.system=all&role.id=588016933bab436d8b7c3ac76827356d => generated 510 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:42:33.647051 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce63ab0-0dee-4c4d-baf3-a50fd7ee6e68 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:33.647621 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce63ab0-0dee-4c4d-baf3-a50fd7ee6e68 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:33.647621 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce63ab0-0dee-4c4d-baf3-a50fd7ee6e68 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:33.649016 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 166/335] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:42:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:42:33.656755 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b3c7c2-76f9-4180-845e-fa32df87b5f6 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:33.657206 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b3c7c2-76f9-4180-845e-fa32df87b5f6 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:33.657302 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b3c7c2-76f9-4180-845e-fa32df87b5f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:33.683684 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-50b3c7c2-76f9-4180-845e-fa32df87b5f6 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:42:33.684873 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-50b3c7c2-76f9-4180-845e-fa32df87b5f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:42:33.760915 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 170/336] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:42:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 09 00:42:34.140418 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bce63ab0-0dee-4c4d-baf3-a50fd7ee6e68 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:34.252720 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bce63ab0-0dee-4c4d-baf3-a50fd7ee6e68 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:34.276956 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5d8ccd5-72f8-4a77-8a15-ffdce80a1577 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:34.278402 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5d8ccd5-72f8-4a77-8a15-ffdce80a1577 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:34.280653 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5d8ccd5-72f8-4a77-8a15-ffdce80a1577 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:34.282222 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 171/337] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:42:34 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 09 00:42:34.288033 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ac2c728-8738-437d-ba3c-e9b31333363b None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:34.288587 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ac2c728-8738-437d-ba3c-e9b31333363b None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:34.288866 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ac2c728-8738-437d-ba3c-e9b31333363b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:34.303556 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94206475-b741-453d-968b-f56dae368154 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:34.304800 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94206475-b741-453d-968b-f56dae368154 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:34.305308 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94206475-b741-453d-968b-f56dae368154 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:34.306502 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94206475-b741-453d-968b-f56dae368154 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:34.327476 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94206475-b741-453d-968b-f56dae368154 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:42:34.327673 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-1ac2c728-8738-437d-ba3c-e9b31333363b None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:42:34.328757 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-1ac2c728-8738-437d-ba3c-e9b31333363b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:42:34.333799 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-94206475-b741-453d-968b-f56dae368154 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 09 00:42:34.333799 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:42:34.333799 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:42:34.333799 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:42:34.333799 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:42:34.333799 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:42:34.333799 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:42:34.333799 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:42:34.333799 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:42:34.333799 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:42:34.333799 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:42:34.333799 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:42:34.333799 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:42:34.333799 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:42:34.333799 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:42:34.333799 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:42:34.334636 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:42:34.334636 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:42:34.334636 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:42:34.334636 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:42:34.334636 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:42:34.334636 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:42:34.334636 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:42:34.334636 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:42:34.334636 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:42:34.334636 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:42:34.334636 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:42:34.334636 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:42:34.334636 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:42:34.334636 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:42:34.334636 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:42:34.334636 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:42:34.335395 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:42:34.335395 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:42:34.335395 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:42:34.335395 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:42:34.335395 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:42:34.335395 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:42:34.335395 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:42:34.335395 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:42:34.335395 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:42:34.335395 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:42:34.335395 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 09 00:42:34.335395 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:42:34.338623 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 167/338] 10.222.0.33 () {56 vars in 1099 bytes} [Sun Oct 9 00:42:34 2022] GET /identity/v3/roles/member => generated 84 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:42:34.354913 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94206475-b741-453d-968b-f56dae368154 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:34.386717 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:34.409990 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:34.438082 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 172/339] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:42:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 09 00:42:34.508511 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94206475-b741-453d-968b-f56dae368154 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:34.556999 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c24bfa1e-bba3-47b2-8023-fcbb8ed51d79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:34.557617 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c24bfa1e-bba3-47b2-8023-fcbb8ed51d79 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:34.557896 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c24bfa1e-bba3-47b2-8023-fcbb8ed51d79 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:34.558169 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c24bfa1e-bba3-47b2-8023-fcbb8ed51d79 None admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:34.558839 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c24bfa1e-bba3-47b2-8023-fcbb8ed51d79 None admin] RBAC: Adding query filter params (name=member) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:34.559141 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c24bfa1e-bba3-47b2-8023-fcbb8ed51d79 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:34.560602 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c24bfa1e-bba3-47b2-8023-fcbb8ed51d79 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:34.572892 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 168/340] 10.222.0.33 () {56 vars in 1101 bytes} [Sun Oct 9 00:42:34 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:34.576050 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ac2c728-8738-437d-ba3c-e9b31333363b None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:34.694756 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ac2c728-8738-437d-ba3c-e9b31333363b None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:34.736659 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43785f59-2399-445a-9aac-efc8d858e3f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:34.737279 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43785f59-2399-445a-9aac-efc8d858e3f8 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:34.737556 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43785f59-2399-445a-9aac-efc8d858e3f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:34.737827 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43785f59-2399-445a-9aac-efc8d858e3f8 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:34.791914 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:34.812887 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43785f59-2399-445a-9aac-efc8d858e3f8 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:34.814410 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43785f59-2399-445a-9aac-efc8d858e3f8 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:34.931480 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 173/341] 10.222.0.33 () {58 vars in 1282 bytes} [Sun Oct 9 00:42:34 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 356 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 09 00:42:34.937666 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c24bfa1e-bba3-47b2-8023-fcbb8ed51d79 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:35.036260 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:35.057970 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c24bfa1e-bba3-47b2-8023-fcbb8ed51d79 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:35.062095 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487baf76-6120-4efa-8943-84450890edb5 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:35.062378 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487baf76-6120-4efa-8943-84450890edb5 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:35.062656 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487baf76-6120-4efa-8943-84450890edb5 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:35.064263 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 174/342] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:42:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:42:35.067234 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-487baf76-6120-4efa-8943-84450890edb5 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:35.135311 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-116779a0-892b-4e3a-8a40-f005fd6f2501 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:35.136619 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-116779a0-892b-4e3a-8a40-f005fd6f2501 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:35.136898 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-116779a0-892b-4e3a-8a40-f005fd6f2501 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:35.137175 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-116779a0-892b-4e3a-8a40-f005fd6f2501 None admin] PATH_INFO: `/v3/projects/768d646302f443f1a7d490b8c3d4bdbb` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:35.138774 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-116779a0-892b-4e3a-8a40-f005fd6f2501 None admin] RBAC: Authorizing `identity:get_project(project_id=768d646302f443f1a7d490b8c3d4bdbb)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:35.140295 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-116779a0-892b-4e3a-8a40-f005fd6f2501 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:35.150365 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 169/343] 10.222.0.33 () {56 vars in 1186 bytes} [Sun Oct 9 00:42:34 2022] GET /identity/v3/projects/768d646302f443f1a7d490b8c3d4bdbb => generated 298 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:35.155811 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88ad4cbf-663c-4c3e-aa13-5fbc30869249 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:35.156099 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88ad4cbf-663c-4c3e-aa13-5fbc30869249 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:35.156381 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88ad4cbf-663c-4c3e-aa13-5fbc30869249 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:35.196369 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-88ad4cbf-663c-4c3e-aa13-5fbc30869249 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:42:35.197511 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-88ad4cbf-663c-4c3e-aa13-5fbc30869249 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:42:35.226913 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:35.266671 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-487baf76-6120-4efa-8943-84450890edb5 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:35.309269 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 170/344] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:42:35 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 09 00:42:35.310646 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88ad4cbf-663c-4c3e-aa13-5fbc30869249 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:35.312987 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-507c38bd-c536-4b7f-8a12-411caaa27001 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:35.313735 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-507c38bd-c536-4b7f-8a12-411caaa27001 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:35.314166 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-507c38bd-c536-4b7f-8a12-411caaa27001 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:35.314532 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-507c38bd-c536-4b7f-8a12-411caaa27001 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:35.326051 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-507c38bd-c536-4b7f-8a12-411caaa27001 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:42:35.331630 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-507c38bd-c536-4b7f-8a12-411caaa27001 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 09 00:42:35.331630 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:42:35.331630 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:42:35.331630 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:42:35.331630 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:42:35.331630 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:42:35.331630 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:42:35.331630 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:42:35.331630 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:42:35.331630 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:42:35.331630 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:42:35.331630 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:42:35.331630 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:42:35.331630 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:42:35.331630 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:42:35.331630 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:42:35.332488 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:42:35.332488 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:42:35.332488 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:42:35.332488 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:42:35.332488 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:42:35.332488 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:42:35.332488 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:42:35.332488 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:42:35.332488 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:42:35.332488 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:42:35.332488 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:42:35.332488 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:42:35.332488 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:42:35.332488 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:42:35.332488 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:42:35.332488 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:42:35.333240 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:42:35.333240 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:42:35.333240 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:42:35.333240 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:42:35.333240 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:42:35.333240 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:42:35.333240 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:42:35.333240 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:42:35.333240 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:42:35.333240 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:42:35.333240 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 09 00:42:35.333240 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:42:35.335833 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 175/345] 10.222.0.33 () {56 vars in 1114 bytes} [Sun Oct 9 00:42:35 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 269 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:42:35.338444 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f1b72e-d00a-4849-80c7-30299daaeab1 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:35.338947 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f1b72e-d00a-4849-80c7-30299daaeab1 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:35.339480 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f1b72e-d00a-4849-80c7-30299daaeab1 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:35.342142 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 176/346] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:42:35 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:42:35.343133 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46f1b72e-d00a-4849-80c7-30299daaeab1 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:35.404549 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:35.426834 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88ad4cbf-663c-4c3e-aa13-5fbc30869249 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:35.466375 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a35f493-f77c-4535-8ce7-881cd0b34d3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:35.467237 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a35f493-f77c-4535-8ce7-881cd0b34d3c None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:35.467712 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a35f493-f77c-4535-8ce7-881cd0b34d3c None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:35.468124 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a35f493-f77c-4535-8ce7-881cd0b34d3c None admin] PATH_INFO: `/v3/groups/e7758a219cfb46dd9c9560aef5255b8a` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:35.469826 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46f1b72e-d00a-4849-80c7-30299daaeab1 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:35.475758 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a35f493-f77c-4535-8ce7-881cd0b34d3c None admin] RBAC: Authorizing `identity:get_group(group_id=e7758a219cfb46dd9c9560aef5255b8a)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:35.477223 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a35f493-f77c-4535-8ce7-881cd0b34d3c None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:35.479436 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 171/347] 10.222.0.33 () {56 vars in 1180 bytes} [Sun Oct 9 00:42:35 2022] GET /identity/v3/groups/e7758a219cfb46dd9c9560aef5255b8a => generated 226 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:35.481192 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c1e4e1-eb11-41bc-a3e0-b5bb7a69342f None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:35.481477 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c1e4e1-eb11-41bc-a3e0-b5bb7a69342f None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:35.481751 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c1e4e1-eb11-41bc-a3e0-b5bb7a69342f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:35.502736 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-b2c1e4e1-eb11-41bc-a3e0-b5bb7a69342f None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:42:35.504240 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-b2c1e4e1-eb11-41bc-a3e0-b5bb7a69342f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:42:35.512579 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ad48eb0-025c-4552-a4e5-8768a460812d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:35.513482 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ad48eb0-025c-4552-a4e5-8768a460812d None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:35.514086 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ad48eb0-025c-4552-a4e5-8768a460812d None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:35.514559 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ad48eb0-025c-4552-a4e5-8768a460812d None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:35.515568 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ad48eb0-025c-4552-a4e5-8768a460812d None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:35.516195 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ad48eb0-025c-4552-a4e5-8768a460812d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:35.517873 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ad48eb0-025c-4552-a4e5-8768a460812d None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:35.527544 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 177/348] 10.222.0.33 () {56 vars in 1111 bytes} [Sun Oct 9 00:42:35 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:35.528859 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ad48eb0-025c-4552-a4e5-8768a460812d None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:35.532329 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:35.579751 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 172/349] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:42:35 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 09 00:42:35.580844 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2c1e4e1-eb11-41bc-a3e0-b5bb7a69342f None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:35.661663 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:35.679624 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ad48eb0-025c-4552-a4e5-8768a460812d None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:35.689680 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2c1e4e1-eb11-41bc-a3e0-b5bb7a69342f None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:35.727942 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-150586b1-74c8-4ee6-a3bd-2f074b0c01c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:35.728783 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-150586b1-74c8-4ee6-a3bd-2f074b0c01c0 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:35.729187 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-150586b1-74c8-4ee6-a3bd-2f074b0c01c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:35.729658 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-150586b1-74c8-4ee6-a3bd-2f074b0c01c0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:35.736320 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da62a471-94cf-486e-843a-782d5072696f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:35.737044 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da62a471-94cf-486e-843a-782d5072696f None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:35.737424 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da62a471-94cf-486e-843a-782d5072696f None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:35.737797 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da62a471-94cf-486e-843a-782d5072696f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:35.738628 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da62a471-94cf-486e-843a-782d5072696f None admin] RBAC: Adding query filter params (group.id=e7758a219cfb46dd9c9560aef5255b8a, role.id=588016933bab436d8b7c3ac76827356d, scope.project.id=768d646302f443f1a7d490b8c3d4bdbb) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:35.739051 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da62a471-94cf-486e-843a-782d5072696f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:35.740802 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da62a471-94cf-486e-843a-782d5072696f None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:35.752580 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 178/350] 10.222.0.33 () {56 vars in 1376 bytes} [Sun Oct 9 00:42:35 2022] GET /identity/v3/role_assignments?group.id=e7758a219cfb46dd9c9560aef5255b8a&scope.project.id=768d646302f443f1a7d490b8c3d4bdbb&role.id=588016933bab436d8b7c3ac76827356d => generated 260 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:35.753600 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da62a471-94cf-486e-843a-782d5072696f None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:35.807956 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-150586b1-74c8-4ee6-a3bd-2f074b0c01c0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:35.810523 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-150586b1-74c8-4ee6-a3bd-2f074b0c01c0 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:35.868846 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:35.896504 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:35.931172 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 173/351] 10.222.0.33 () {58 vars in 1282 bytes} [Sun Oct 9 00:42:35 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 350 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 09 00:42:35.937017 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-150586b1-74c8-4ee6-a3bd-2f074b0c01c0 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:35.945051 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da62a471-94cf-486e-843a-782d5072696f None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:36.014321 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f728baed-b1a1-440e-ac63-1dbf76f554ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:36.015011 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f728baed-b1a1-440e-ac63-1dbf76f554ad None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:36.015291 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f728baed-b1a1-440e-ac63-1dbf76f554ad None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:36.015564 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f728baed-b1a1-440e-ac63-1dbf76f554ad None admin] PATH_INFO: `/v3/roles/member` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:36.034431 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f728baed-b1a1-440e-ac63-1dbf76f554ad None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:42:36.045319 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-f728baed-b1a1-440e-ac63-1dbf76f554ad None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 09 00:42:36.045319 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:42:36.045319 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:42:36.045319 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:42:36.045319 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:42:36.045319 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:42:36.045319 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:42:36.045319 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:42:36.045319 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:42:36.045319 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:42:36.045319 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:42:36.045319 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:42:36.045319 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:42:36.045319 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:42:36.045319 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:42:36.045319 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:42:36.046256 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:42:36.046256 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:42:36.046256 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:42:36.046256 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:42:36.046256 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:42:36.046256 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:42:36.046256 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:42:36.046256 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:42:36.046256 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:42:36.046256 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:42:36.046256 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:42:36.046256 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:42:36.046256 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:42:36.046256 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:42:36.046256 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:42:36.046256 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:42:36.047035 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:42:36.047035 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:42:36.047035 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:42:36.047035 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:42:36.047035 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:42:36.047035 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:42:36.047035 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:42:36.047035 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:42:36.047035 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:42:36.047035 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:42:36.047035 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 09 00:42:36.047035 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:42:36.047035 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 179/352] 10.222.0.33 () {56 vars in 1099 bytes} [Sun Oct 9 00:42:35 2022] GET /identity/v3/roles/member => generated 84 bytes in 293 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:42:36.047717 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86843fcb-061f-41b6-920b-411d29bee75b None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:36.051986 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86843fcb-061f-41b6-920b-411d29bee75b None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:36.052272 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86843fcb-061f-41b6-920b-411d29bee75b None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:36.053773 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 180/353] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:42:36 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:42:36.060828 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86843fcb-061f-41b6-920b-411d29bee75b None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:36.091158 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-150586b1-74c8-4ee6-a3bd-2f074b0c01c0 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:36.132687 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54cb6c2d-8e62-4d9a-9812-78f5f8b9e24d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:36.134279 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54cb6c2d-8e62-4d9a-9812-78f5f8b9e24d None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:36.134760 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54cb6c2d-8e62-4d9a-9812-78f5f8b9e24d None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:36.135230 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54cb6c2d-8e62-4d9a-9812-78f5f8b9e24d None admin] PATH_INFO: `/v3/projects/768d646302f443f1a7d490b8c3d4bdbb` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:36.137321 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54cb6c2d-8e62-4d9a-9812-78f5f8b9e24d None admin] RBAC: Authorizing `identity:get_project(project_id=768d646302f443f1a7d490b8c3d4bdbb)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:36.139042 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54cb6c2d-8e62-4d9a-9812-78f5f8b9e24d None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:36.141106 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 174/354] 10.222.0.33 () {56 vars in 1186 bytes} [Sun Oct 9 00:42:35 2022] GET /identity/v3/projects/768d646302f443f1a7d490b8c3d4bdbb => generated 298 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:36.142486 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de9926c-7df6-4921-9c19-20403eed71e5 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:36.142973 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de9926c-7df6-4921-9c19-20403eed71e5 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:36.143459 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de9926c-7df6-4921-9c19-20403eed71e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:36.163091 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-4de9926c-7df6-4921-9c19-20403eed71e5 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:42:36.164984 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-4de9926c-7df6-4921-9c19-20403eed71e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:42:36.205532 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:36.248183 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 175/355] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:42:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 09 00:42:36.249229 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4de9926c-7df6-4921-9c19-20403eed71e5 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:36.252985 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86843fcb-061f-41b6-920b-411d29bee75b None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:36.281782 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:36.292628 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b702c66-e083-4614-b97c-43deba398af3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:36.293528 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b702c66-e083-4614-b97c-43deba398af3 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:36.293988 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b702c66-e083-4614-b97c-43deba398af3 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:36.294464 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b702c66-e083-4614-b97c-43deba398af3 None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:36.295304 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b702c66-e083-4614-b97c-43deba398af3 None admin] RBAC: Adding query filter params (name=member) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:36.295811 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b702c66-e083-4614-b97c-43deba398af3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:36.297367 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b702c66-e083-4614-b97c-43deba398af3 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:36.307561 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 181/356] 10.222.0.33 () {56 vars in 1101 bytes} [Sun Oct 9 00:42:36 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:36.308666 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b702c66-e083-4614-b97c-43deba398af3 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:36.416865 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4de9926c-7df6-4921-9c19-20403eed71e5 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:36.417921 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b702c66-e083-4614-b97c-43deba398af3 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:36.455206 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a420d49f-ffc1-480c-85ed-d2076d78e97e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:36.455870 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a420d49f-ffc1-480c-85ed-d2076d78e97e None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:36.456154 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a420d49f-ffc1-480c-85ed-d2076d78e97e None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:36.456430 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a420d49f-ffc1-480c-85ed-d2076d78e97e None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:36.468048 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a420d49f-ffc1-480c-85ed-d2076d78e97e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:42:36.473434 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-a420d49f-ffc1-480c-85ed-d2076d78e97e None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 09 00:42:36.473434 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:42:36.473434 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:42:36.473434 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:42:36.473434 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:42:36.473434 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:42:36.473434 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:42:36.473434 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:42:36.473434 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:42:36.473434 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:42:36.473434 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:42:36.473434 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:42:36.473434 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:42:36.473434 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:42:36.473434 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:42:36.473434 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:42:36.474516 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:42:36.474516 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:42:36.474516 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:42:36.474516 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:42:36.474516 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:42:36.474516 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:42:36.474516 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:42:36.474516 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:42:36.474516 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:42:36.474516 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:42:36.474516 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:42:36.474516 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:42:36.474516 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:42:36.474516 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:42:36.474516 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:42:36.474516 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:42:36.475499 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:42:36.475499 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:42:36.475499 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:42:36.475499 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:42:36.475499 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:42:36.475499 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:42:36.475499 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:42:36.475499 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:42:36.475499 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:42:36.475499 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:42:36.475499 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 09 00:42:36.475499 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:42:36.481295 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 182/357] 10.222.0.33 () {56 vars in 1099 bytes} [Sun Oct 9 00:42:36 2022] GET /identity/v3/roles/reader => generated 84 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:42:36.482337 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a420d49f-ffc1-480c-85ed-d2076d78e97e None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:36.484837 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d167fd88-6009-47da-b32d-221d504f356f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:36.485497 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d167fd88-6009-47da-b32d-221d504f356f None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:36.485836 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d167fd88-6009-47da-b32d-221d504f356f None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:36.486111 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d167fd88-6009-47da-b32d-221d504f356f None admin] PATH_INFO: `/v3/groups/e7758a219cfb46dd9c9560aef5255b8a` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:36.487377 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d167fd88-6009-47da-b32d-221d504f356f None admin] RBAC: Authorizing `identity:get_group(group_id=e7758a219cfb46dd9c9560aef5255b8a)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:36.489097 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d167fd88-6009-47da-b32d-221d504f356f None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:36.490940 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 176/358] 10.222.0.33 () {56 vars in 1180 bytes} [Sun Oct 9 00:42:36 2022] GET /identity/v3/groups/e7758a219cfb46dd9c9560aef5255b8a => generated 226 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:36.497448 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d167fd88-6009-47da-b32d-221d504f356f None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:36.540995 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:36.572716 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:36.599224 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a420d49f-ffc1-480c-85ed-d2076d78e97e None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:36.639504 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1991570-17b6-409c-a8fb-76dce85d52ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:36.640256 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1991570-17b6-409c-a8fb-76dce85d52ae None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:36.640550 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1991570-17b6-409c-a8fb-76dce85d52ae None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:36.642342 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1991570-17b6-409c-a8fb-76dce85d52ae None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:36.655615 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d167fd88-6009-47da-b32d-221d504f356f None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:36.698874 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-036c583f-0de1-48ed-a3cc-59d2a7aad7a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:36.699899 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-036c583f-0de1-48ed-a3cc-59d2a7aad7a8 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:36.700471 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-036c583f-0de1-48ed-a3cc-59d2a7aad7a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:36.700941 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-036c583f-0de1-48ed-a3cc-59d2a7aad7a8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:36.701941 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-036c583f-0de1-48ed-a3cc-59d2a7aad7a8 None admin] RBAC: Adding query filter params (group.id=e7758a219cfb46dd9c9560aef5255b8a, role.id=7c1fdac272a34be99661077b0f1b8014, scope.project.id=768d646302f443f1a7d490b8c3d4bdbb) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:36.702565 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-036c583f-0de1-48ed-a3cc-59d2a7aad7a8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:36.704320 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-036c583f-0de1-48ed-a3cc-59d2a7aad7a8 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:36.710655 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 177/359] 10.222.0.33 () {56 vars in 1376 bytes} [Sun Oct 9 00:42:36 2022] GET /identity/v3/role_assignments?group.id=e7758a219cfb46dd9c9560aef5255b8a&scope.project.id=768d646302f443f1a7d490b8c3d4bdbb&role.id=7c1fdac272a34be99661077b0f1b8014 => generated 260 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:36.711785 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-036c583f-0de1-48ed-a3cc-59d2a7aad7a8 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:36.716263 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1991570-17b6-409c-a8fb-76dce85d52ae None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:36.717776 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1991570-17b6-409c-a8fb-76dce85d52ae None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:36.785942 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:36.862028 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-036c583f-0de1-48ed-a3cc-59d2a7aad7a8 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:36.864808 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:36.872667 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 183/360] 10.222.0.33 () {58 vars in 1282 bytes} [Sun Oct 9 00:42:36 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 390 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 09 00:42:36.878070 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1991570-17b6-409c-a8fb-76dce85d52ae None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:36.910172 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3b90ae1-b541-4d5a-9d42-dca1bda637ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:36.911183 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3b90ae1-b541-4d5a-9d42-dca1bda637ba None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:36.911721 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3b90ae1-b541-4d5a-9d42-dca1bda637ba None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:36.912194 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3b90ae1-b541-4d5a-9d42-dca1bda637ba None admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:36.913064 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3b90ae1-b541-4d5a-9d42-dca1bda637ba None admin] RBAC: Adding query filter params (name=reader) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:36.913824 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3b90ae1-b541-4d5a-9d42-dca1bda637ba None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:36.916025 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3b90ae1-b541-4d5a-9d42-dca1bda637ba None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:36.925841 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 178/361] 10.222.0.33 () {56 vars in 1101 bytes} [Sun Oct 9 00:42:36 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:36.927460 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-393956f4-9167-4b91-9c9c-c5131912f749 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:36.927971 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-393956f4-9167-4b91-9c9c-c5131912f749 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:36.928507 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-393956f4-9167-4b91-9c9c-c5131912f749 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:36.930328 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 179/362] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:42:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:42:36.931333 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-393956f4-9167-4b91-9c9c-c5131912f749 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:36.994658 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1991570-17b6-409c-a8fb-76dce85d52ae None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:37.036347 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39cd4930-5992-4331-8c1a-2fb37be438e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:37.038597 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39cd4930-5992-4331-8c1a-2fb37be438e9 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:37.038597 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39cd4930-5992-4331-8c1a-2fb37be438e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:37.038597 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39cd4930-5992-4331-8c1a-2fb37be438e9 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:37.049913 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-393956f4-9167-4b91-9c9c-c5131912f749 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:37.050421 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39cd4930-5992-4331-8c1a-2fb37be438e9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:42:37.055447 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-39cd4930-5992-4331-8c1a-2fb37be438e9 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 09 00:42:37.055447 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:42:37.055447 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:42:37.055447 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:42:37.055447 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:42:37.055447 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:42:37.055447 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:42:37.055447 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:42:37.055447 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:42:37.055447 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:42:37.055447 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:42:37.055447 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:42:37.055447 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:42:37.055447 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:42:37.055447 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:42:37.055447 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:42:37.056337 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:42:37.056337 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:42:37.056337 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:42:37.056337 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:42:37.056337 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:42:37.056337 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:42:37.056337 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:42:37.056337 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:42:37.056337 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:42:37.056337 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:42:37.056337 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:42:37.056337 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:42:37.056337 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:42:37.056337 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:42:37.056337 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:42:37.056337 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:42:37.057115 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:42:37.057115 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:42:37.057115 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:42:37.057115 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:42:37.057115 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:42:37.057115 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:42:37.057115 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:42:37.057115 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:42:37.057115 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:42:37.057115 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:42:37.057115 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 09 00:42:37.057115 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:42:37.057115 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 184/363] 10.222.0.33 () {56 vars in 1114 bytes} [Sun Oct 9 00:42:36 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:42:37.058240 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ecf5bc5-a5b6-40e2-be1c-a6c74f7f84b2 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:37.058240 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ecf5bc5-a5b6-40e2-be1c-a6c74f7f84b2 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:37.058463 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ecf5bc5-a5b6-40e2-be1c-a6c74f7f84b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:37.079440 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-9ecf5bc5-a5b6-40e2-be1c-a6c74f7f84b2 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:42:37.081134 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-9ecf5bc5-a5b6-40e2-be1c-a6c74f7f84b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:42:37.082968 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:37.092968 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4441e523-fb4f-4b59-9673-2dae116f2791 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:37.095310 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4441e523-fb4f-4b59-9673-2dae116f2791 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:37.095832 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4441e523-fb4f-4b59-9673-2dae116f2791 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:37.096307 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4441e523-fb4f-4b59-9673-2dae116f2791 None admin] PATH_INFO: `/v3/projects/dd8011750cb34075a6f8106533f848c2` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:37.097661 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4441e523-fb4f-4b59-9673-2dae116f2791 None admin] RBAC: Authorizing `identity:get_project(project_id=dd8011750cb34075a6f8106533f848c2)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:37.099556 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4441e523-fb4f-4b59-9673-2dae116f2791 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:37.101544 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 180/364] 10.222.0.33 () {56 vars in 1186 bytes} [Sun Oct 9 00:42:36 2022] GET /identity/v3/projects/dd8011750cb34075a6f8106533f848c2 => generated 302 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:37.103249 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4441e523-fb4f-4b59-9673-2dae116f2791 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:37.145364 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:37.163186 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 185/365] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:42:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 09 00:42:37.164590 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ecf5bc5-a5b6-40e2-be1c-a6c74f7f84b2 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:37.219004 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4441e523-fb4f-4b59-9673-2dae116f2791 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:37.255748 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26fba50d-e725-4337-b882-7789eec4d2a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:37.256792 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26fba50d-e725-4337-b882-7789eec4d2a0 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:37.257272 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26fba50d-e725-4337-b882-7789eec4d2a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:37.257741 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26fba50d-e725-4337-b882-7789eec4d2a0 None admin] PATH_INFO: `/v3/users/ea9a35b7be1c4625a0f385ee980e96ea` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:37.259757 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26fba50d-e725-4337-b882-7789eec4d2a0 None admin] RBAC: Authorizing `identity:get_user(user_id=ea9a35b7be1c4625a0f385ee980e96ea)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:37.261565 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26fba50d-e725-4337-b882-7789eec4d2a0 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:37.263374 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 181/366] 10.222.0.33 () {56 vars in 1177 bytes} [Sun Oct 9 00:42:37 2022] GET /identity/v3/users/ea9a35b7be1c4625a0f385ee980e96ea => generated 293 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:37.264335 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26fba50d-e725-4337-b882-7789eec4d2a0 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:37.285552 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ecf5bc5-a5b6-40e2-be1c-a6c74f7f84b2 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:37.331510 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0548d98-c409-42f5-903a-3eae76563427 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:37.332218 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0548d98-c409-42f5-903a-3eae76563427 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:37.333647 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0548d98-c409-42f5-903a-3eae76563427 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:37.333647 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0548d98-c409-42f5-903a-3eae76563427 None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:37.333810 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0548d98-c409-42f5-903a-3eae76563427 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:37.334110 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0548d98-c409-42f5-903a-3eae76563427 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:37.336171 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0548d98-c409-42f5-903a-3eae76563427 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:37.343782 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:37.350529 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 186/367] 10.222.0.33 () {56 vars in 1111 bytes} [Sun Oct 9 00:42:37 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:37.351848 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0548d98-c409-42f5-903a-3eae76563427 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:37.394440 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26fba50d-e725-4337-b882-7789eec4d2a0 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:37.397768 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:37.438252 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75185c8b-81a3-4436-aa8d-f0f172090f1c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:37.439649 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75185c8b-81a3-4436-aa8d-f0f172090f1c None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:37.440116 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75185c8b-81a3-4436-aa8d-f0f172090f1c None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:37.440774 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75185c8b-81a3-4436-aa8d-f0f172090f1c None admin] PATH_INFO: `/v3/groups/e7758a219cfb46dd9c9560aef5255b8a` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:37.442027 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75185c8b-81a3-4436-aa8d-f0f172090f1c None admin] RBAC: Authorizing `identity:get_group(group_id=e7758a219cfb46dd9c9560aef5255b8a)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:37.443664 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75185c8b-81a3-4436-aa8d-f0f172090f1c None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:37.445730 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 182/368] 10.222.0.33 () {56 vars in 1180 bytes} [Sun Oct 9 00:42:37 2022] GET /identity/v3/groups/e7758a219cfb46dd9c9560aef5255b8a => generated 226 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:37.447276 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75185c8b-81a3-4436-aa8d-f0f172090f1c None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:37.487694 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0548d98-c409-42f5-903a-3eae76563427 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:37.525304 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37095e67-31d0-4e82-97b8-83257dfb883e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:37.526235 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37095e67-31d0-4e82-97b8-83257dfb883e None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:37.526738 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37095e67-31d0-4e82-97b8-83257dfb883e None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:37.527210 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37095e67-31d0-4e82-97b8-83257dfb883e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:37.528158 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37095e67-31d0-4e82-97b8-83257dfb883e None admin] RBAC: Adding query filter params (role.id=2d631b6eec1646c9834f74efa1d66091, user.id=ea9a35b7be1c4625a0f385ee980e96ea, scope.system=all) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:37.528678 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37095e67-31d0-4e82-97b8-83257dfb883e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:37.530347 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37095e67-31d0-4e82-97b8-83257dfb883e None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:37.545924 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 187/369] 10.222.0.33 () {56 vars in 1308 bytes} [Sun Oct 9 00:42:37 2022] GET /identity/v3/role_assignments?user.id=ea9a35b7be1c4625a0f385ee980e96ea&scope.system=all&role.id=2d631b6eec1646c9834f74efa1d66091 => generated 510 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:37.547086 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37095e67-31d0-4e82-97b8-83257dfb883e None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:37.579823 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75185c8b-81a3-4436-aa8d-f0f172090f1c None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:37.636273 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13e38a62-4d14-4e43-b1fd-2d082125c6dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:37.636963 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e38a62-4d14-4e43-b1fd-2d082125c6dd None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:37.637248 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e38a62-4d14-4e43-b1fd-2d082125c6dd None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:37.637536 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e38a62-4d14-4e43-b1fd-2d082125c6dd None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:37.672506 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:37.704137 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:37.764022 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13e38a62-4d14-4e43-b1fd-2d082125c6dd None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:37.766035 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13e38a62-4d14-4e43-b1fd-2d082125c6dd None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:37.770807 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37095e67-31d0-4e82-97b8-83257dfb883e None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:37.840150 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad9cdd07-c63b-4bda-b09c-6b41e05052f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:37.841559 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9cdd07-c63b-4bda-b09c-6b41e05052f7 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:37.842058 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9cdd07-c63b-4bda-b09c-6b41e05052f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:37.842589 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9cdd07-c63b-4bda-b09c-6b41e05052f7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:37.843658 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad9cdd07-c63b-4bda-b09c-6b41e05052f7 None admin] RBAC: Adding query filter params (group.id=e7758a219cfb46dd9c9560aef5255b8a, role.id=588016933bab436d8b7c3ac76827356d, scope.project.id=dd8011750cb34075a6f8106533f848c2) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:37.844179 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad9cdd07-c63b-4bda-b09c-6b41e05052f7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:37.845826 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad9cdd07-c63b-4bda-b09c-6b41e05052f7 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:37.864929 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 188/370] 10.222.0.33 () {56 vars in 1376 bytes} [Sun Oct 9 00:42:37 2022] GET /identity/v3/role_assignments?group.id=e7758a219cfb46dd9c9560aef5255b8a&scope.project.id=dd8011750cb34075a6f8106533f848c2&role.id=588016933bab436d8b7c3ac76827356d => generated 260 bytes in 318 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:37.866207 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad9cdd07-c63b-4bda-b09c-6b41e05052f7 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:37.976732 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 183/371] 10.222.0.33 () {58 vars in 1282 bytes} [Sun Oct 9 00:42:37 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 528 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 09 00:42:37.980682 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13e38a62-4d14-4e43-b1fd-2d082125c6dd None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:38.055119 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad9cdd07-c63b-4bda-b09c-6b41e05052f7 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:38.091711 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:38.107157 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d8151d0-ca4f-4a16-954c-44fca9c76f49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:38.107831 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8151d0-ca4f-4a16-954c-44fca9c76f49 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:38.108112 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8151d0-ca4f-4a16-954c-44fca9c76f49 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:38.108384 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8151d0-ca4f-4a16-954c-44fca9c76f49 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:38.134218 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d8151d0-ca4f-4a16-954c-44fca9c76f49 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:42:38.139651 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-6d8151d0-ca4f-4a16-954c-44fca9c76f49 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 09 00:42:38.139651 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:42:38.139651 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:42:38.139651 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:42:38.139651 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:42:38.139651 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:42:38.139651 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:42:38.139651 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:42:38.139651 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:42:38.139651 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:42:38.139651 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:42:38.139651 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:42:38.139651 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:42:38.139651 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:42:38.139651 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:42:38.139651 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:42:38.140666 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:42:38.140666 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:42:38.140666 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:42:38.140666 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:42:38.140666 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:42:38.140666 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:42:38.140666 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:42:38.140666 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:42:38.140666 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:42:38.140666 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:42:38.140666 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:42:38.140666 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:42:38.140666 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:42:38.140666 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:42:38.140666 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:42:38.140666 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:42:38.141478 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:42:38.141478 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:42:38.141478 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:42:38.141478 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:42:38.141478 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:42:38.141478 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:42:38.141478 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:42:38.141478 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:42:38.141478 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:42:38.141478 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:42:38.141478 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 09 00:42:38.141478 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:42:38.143547 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 189/372] 10.222.0.33 () {56 vars in 1096 bytes} [Sun Oct 9 00:42:37 2022] GET /identity/v3/roles/admin => generated 83 bytes in 277 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:42:38.152438 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d8151d0-ca4f-4a16-954c-44fca9c76f49 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:38.177169 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:38.216385 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13e38a62-4d14-4e43-b1fd-2d082125c6dd None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:38.254544 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d0c0b6f-d86f-4325-8b38-98c0d0ebc1c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:38.255181 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d0c0b6f-d86f-4325-8b38-98c0d0ebc1c7 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:38.255457 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d0c0b6f-d86f-4325-8b38-98c0d0ebc1c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:38.255764 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d0c0b6f-d86f-4325-8b38-98c0d0ebc1c7 None admin] PATH_INFO: `/v3/projects/dd8011750cb34075a6f8106533f848c2` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:38.256958 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d0c0b6f-d86f-4325-8b38-98c0d0ebc1c7 None admin] RBAC: Authorizing `identity:get_project(project_id=dd8011750cb34075a6f8106533f848c2)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:38.258552 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d0c0b6f-d86f-4325-8b38-98c0d0ebc1c7 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:38.260573 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 184/373] 10.222.0.33 () {56 vars in 1186 bytes} [Sun Oct 9 00:42:37 2022] GET /identity/v3/projects/dd8011750cb34075a6f8106533f848c2 => generated 302 bytes in 280 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:42:38.265651 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d0c0b6f-d86f-4325-8b38-98c0d0ebc1c7 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:38.285461 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d8151d0-ca4f-4a16-954c-44fca9c76f49 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:38.332348 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a773ac8-f43b-45b3-a92c-c252ff581437 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:38.332977 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a773ac8-f43b-45b3-a92c-c252ff581437 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:38.333264 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a773ac8-f43b-45b3-a92c-c252ff581437 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:38.333537 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a773ac8-f43b-45b3-a92c-c252ff581437 None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:38.334213 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a773ac8-f43b-45b3-a92c-c252ff581437 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:38.334513 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a773ac8-f43b-45b3-a92c-c252ff581437 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:38.336023 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a773ac8-f43b-45b3-a92c-c252ff581437 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:38.345771 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 190/374] 10.222.0.33 () {56 vars in 1099 bytes} [Sun Oct 9 00:42:38 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:42:38.357345 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a773ac8-f43b-45b3-a92c-c252ff581437 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:38.400439 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d0c0b6f-d86f-4325-8b38-98c0d0ebc1c7 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:38.424633 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:38.437282 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4de207e2-2974-48f1-b576-5647c4a55911 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:38.437924 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de207e2-2974-48f1-b576-5647c4a55911 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:38.438203 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de207e2-2974-48f1-b576-5647c4a55911 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:38.438475 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de207e2-2974-48f1-b576-5647c4a55911 None admin] PATH_INFO: `/v3/groups/e7758a219cfb46dd9c9560aef5255b8a` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:38.439608 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4de207e2-2974-48f1-b576-5647c4a55911 None admin] RBAC: Authorizing `identity:get_group(group_id=e7758a219cfb46dd9c9560aef5255b8a)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:38.441065 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4de207e2-2974-48f1-b576-5647c4a55911 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:38.442942 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 185/375] 10.222.0.33 () {56 vars in 1180 bytes} [Sun Oct 9 00:42:38 2022] GET /identity/v3/groups/e7758a219cfb46dd9c9560aef5255b8a => generated 226 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:38.448345 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4de207e2-2974-48f1-b576-5647c4a55911 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:38.457097 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:38.507662 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a773ac8-f43b-45b3-a92c-c252ff581437 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:38.557466 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a17e414e-59a1-468e-8854-2c17bed7d5d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:38.558087 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a17e414e-59a1-468e-8854-2c17bed7d5d0 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:38.558365 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a17e414e-59a1-468e-8854-2c17bed7d5d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:38.558640 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a17e414e-59a1-468e-8854-2c17bed7d5d0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:38.614943 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4de207e2-2974-48f1-b576-5647c4a55911 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:38.671419 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd6ec294-144e-4671-aa4f-a1510f0e5f8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:38.672124 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd6ec294-144e-4671-aa4f-a1510f0e5f8f None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:38.672409 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd6ec294-144e-4671-aa4f-a1510f0e5f8f None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:38.672681 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd6ec294-144e-4671-aa4f-a1510f0e5f8f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:38.673414 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd6ec294-144e-4671-aa4f-a1510f0e5f8f None admin] RBAC: Adding query filter params (group.id=e7758a219cfb46dd9c9560aef5255b8a, role.id=7c1fdac272a34be99661077b0f1b8014, scope.project.id=dd8011750cb34075a6f8106533f848c2) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:38.673726 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd6ec294-144e-4671-aa4f-a1510f0e5f8f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:38.675162 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd6ec294-144e-4671-aa4f-a1510f0e5f8f None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:38.688751 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 186/376] 10.222.0.33 () {56 vars in 1376 bytes} [Sun Oct 9 00:42:38 2022] GET /identity/v3/role_assignments?group.id=e7758a219cfb46dd9c9560aef5255b8a&scope.project.id=dd8011750cb34075a6f8106533f848c2&role.id=7c1fdac272a34be99661077b0f1b8014 => generated 260 bytes in 241 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:42:38.691713 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a17e414e-59a1-468e-8854-2c17bed7d5d0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:38.693233 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a17e414e-59a1-468e-8854-2c17bed7d5d0 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:38.778778 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:38.837650 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 191/377] 10.222.0.33 () {58 vars in 1282 bytes} [Sun Oct 9 00:42:38 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 481 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 09 00:42:38.843806 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd6ec294-144e-4671-aa4f-a1510f0e5f8f None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:38.923834 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:38.959427 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd6ec294-144e-4671-aa4f-a1510f0e5f8f None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:39.005499 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11059ee0-bc15-426d-950c-668e1caaf1a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:39.006146 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11059ee0-bc15-426d-950c-668e1caaf1a8 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:39.006430 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11059ee0-bc15-426d-950c-668e1caaf1a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:39.006707 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11059ee0-bc15-426d-950c-668e1caaf1a8 None admin] PATH_INFO: `/v3/projects/34e0fbee16724df3abcac7131a4822aa` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:39.007880 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11059ee0-bc15-426d-950c-668e1caaf1a8 None admin] RBAC: Authorizing `identity:get_project(project_id=34e0fbee16724df3abcac7131a4822aa)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:39.009413 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11059ee0-bc15-426d-950c-668e1caaf1a8 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:39.012401 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 187/378] 10.222.0.33 () {56 vars in 1186 bytes} [Sun Oct 9 00:42:38 2022] GET /identity/v3/projects/34e0fbee16724df3abcac7131a4822aa => generated 344 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:39.017064 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a17e414e-59a1-468e-8854-2c17bed7d5d0 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:39.141114 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a17e414e-59a1-468e-8854-2c17bed7d5d0 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:39.185904 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dadaa33-eeb5-4634-96de-9a3ceac79815 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:39.186458 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dadaa33-eeb5-4634-96de-9a3ceac79815 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:39.186739 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dadaa33-eeb5-4634-96de-9a3ceac79815 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:39.187013 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dadaa33-eeb5-4634-96de-9a3ceac79815 None admin] PATH_INFO: `/v3/groups/3026808ce79d453990adc3fe4e692084` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:39.193300 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b08a10-cd18-49ce-9ef8-6b10e9de47a7 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:39.194215 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b08a10-cd18-49ce-9ef8-6b10e9de47a7 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:39.194695 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b08a10-cd18-49ce-9ef8-6b10e9de47a7 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:39.195186 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dadaa33-eeb5-4634-96de-9a3ceac79815 None admin] RBAC: Authorizing `identity:get_group(group_id=3026808ce79d453990adc3fe4e692084)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:39.197466 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dadaa33-eeb5-4634-96de-9a3ceac79815 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:39.198840 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 188/379] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:42:39 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 09 00:42:39.198946 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 192/380] 10.222.0.33 () {56 vars in 1180 bytes} [Sun Oct 9 00:42:39 2022] GET /identity/v3/groups/3026808ce79d453990adc3fe4e692084 => generated 229 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:39.204291 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-728792a4-2bbc-4b2c-9486-10a9cc396fa5 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:39.204580 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-728792a4-2bbc-4b2c-9486-10a9cc396fa5 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:39.204862 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-728792a4-2bbc-4b2c-9486-10a9cc396fa5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:39.210658 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65b08a10-cd18-49ce-9ef8-6b10e9de47a7 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:39.225501 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-728792a4-2bbc-4b2c-9486-10a9cc396fa5 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:42:39.236160 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-728792a4-2bbc-4b2c-9486-10a9cc396fa5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:42:39.236160 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:39.324950 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 193/381] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:42:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 09 00:42:39.371644 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65b08a10-cd18-49ce-9ef8-6b10e9de47a7 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:39.406807 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1aaf3fe-23c8-4283-8a76-8e62bf0abbff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:39.407466 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1aaf3fe-23c8-4283-8a76-8e62bf0abbff None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:39.407786 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1aaf3fe-23c8-4283-8a76-8e62bf0abbff None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:39.408061 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1aaf3fe-23c8-4283-8a76-8e62bf0abbff None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:39.408840 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1aaf3fe-23c8-4283-8a76-8e62bf0abbff None admin] RBAC: Adding query filter params (group.id=3026808ce79d453990adc3fe4e692084, role.id=092036ee46ac427baf4d56468575f697, scope.project.id=34e0fbee16724df3abcac7131a4822aa) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:39.409151 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1aaf3fe-23c8-4283-8a76-8e62bf0abbff None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:39.410654 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1aaf3fe-23c8-4283-8a76-8e62bf0abbff None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:39.417395 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 189/382] 10.222.0.33 () {56 vars in 1376 bytes} [Sun Oct 9 00:42:39 2022] GET /identity/v3/role_assignments?group.id=3026808ce79d453990adc3fe4e692084&scope.project.id=34e0fbee16724df3abcac7131a4822aa&role.id=092036ee46ac427baf4d56468575f697 => generated 260 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:39.561878 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5960ae2b-3d78-43df-b7e0-6d3ae8f652aa None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:39.562183 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5960ae2b-3d78-43df-b7e0-6d3ae8f652aa None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:39.562353 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5960ae2b-3d78-43df-b7e0-6d3ae8f652aa None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:39.563795 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 194/383] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:42:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:42:39.569253 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-526aea1d-1d3a-4d6d-b549-9c1660e9755a None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:39.569850 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-526aea1d-1d3a-4d6d-b549-9c1660e9755a None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:39.570322 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-526aea1d-1d3a-4d6d-b549-9c1660e9755a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:39.578056 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:39.596630 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-526aea1d-1d3a-4d6d-b549-9c1660e9755a None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:42:39.598190 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-526aea1d-1d3a-4d6d-b549-9c1660e9755a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:42:39.679928 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 190/384] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:42:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 09 00:42:39.752096 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5960ae2b-3d78-43df-b7e0-6d3ae8f652aa None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:39.891286 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5960ae2b-3d78-43df-b7e0-6d3ae8f652aa None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:39.935832 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e6813a2-4782-4c40-a6f0-408af9a233d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:39.936892 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e6813a2-4782-4c40-a6f0-408af9a233d5 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:39.937412 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e6813a2-4782-4c40-a6f0-408af9a233d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:39.937884 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e6813a2-4782-4c40-a6f0-408af9a233d5 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:39.947803 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:39.954679 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e6813a2-4782-4c40-a6f0-408af9a233d5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:42:39.960266 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-0e6813a2-4782-4c40-a6f0-408af9a233d5 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 09 00:42:39.960266 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:42:39.960266 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:42:39.960266 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:42:39.960266 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:42:39.960266 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:42:39.960266 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:42:39.960266 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:42:39.960266 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:42:39.960266 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:42:39.960266 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:42:39.960266 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:42:39.960266 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:42:39.960266 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:42:39.960266 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:42:39.960266 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:42:39.961121 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:42:39.961121 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:42:39.961121 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:42:39.961121 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:42:39.961121 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:42:39.961121 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:42:39.961121 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:42:39.961121 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:42:39.961121 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:42:39.961121 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:42:39.961121 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:42:39.961121 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:42:39.961121 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:42:39.961121 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:42:39.961121 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:42:39.961121 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:42:39.965924 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:42:39.965924 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:42:39.965924 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:42:39.965924 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:42:39.965924 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:42:39.965924 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:42:39.965924 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:42:39.965924 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:42:39.965924 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:42:39.965924 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:42:39.965924 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 09 00:42:39.965924 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:42:39.968695 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 195/385] 10.222.0.33 () {56 vars in 1099 bytes} [Sun Oct 9 00:42:39 2022] GET /identity/v3/roles/member => generated 84 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:42:39.974453 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e6813a2-4782-4c40-a6f0-408af9a233d5 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:39.974606 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-526aea1d-1d3a-4d6d-b549-9c1660e9755a None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:40.130928 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-526aea1d-1d3a-4d6d-b549-9c1660e9755a None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:40.160775 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e6813a2-4782-4c40-a6f0-408af9a233d5 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:40.169030 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:40.180081 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c71e9ad-6203-4f7a-919f-89b5342b623f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:40.180803 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c71e9ad-6203-4f7a-919f-89b5342b623f None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:40.181190 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c71e9ad-6203-4f7a-919f-89b5342b623f None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:40.181468 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c71e9ad-6203-4f7a-919f-89b5342b623f None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:40.197590 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c71e9ad-6203-4f7a-919f-89b5342b623f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:42:40.202414 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd50dbc5-e955-4f33-8d02-74d8cda5fbf2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:40.202652 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-3c71e9ad-6203-4f7a-919f-89b5342b623f None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 09 00:42:40.202652 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:42:40.202652 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:42:40.202652 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:42:40.202652 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:42:40.202652 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:42:40.202652 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:42:40.202652 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:42:40.202652 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:42:40.202652 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:42:40.202652 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:42:40.202652 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:42:40.202652 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:42:40.202652 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:42:40.202652 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:42:40.202652 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:42:40.203615 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:42:40.203615 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:42:40.203615 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:42:40.203615 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:42:40.203615 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:42:40.203615 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:42:40.203615 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:42:40.203615 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:42:40.203615 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:42:40.203615 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:42:40.203615 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:42:40.203615 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:42:40.203615 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:42:40.203615 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:42:40.203615 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:42:40.203615 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:42:40.204420 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:42:40.204420 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:42:40.204420 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:42:40.204420 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:42:40.204420 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:42:40.204420 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:42:40.204420 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:42:40.204420 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:42:40.204420 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:42:40.204420 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:42:40.204420 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 09 00:42:40.204420 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:42:40.204420 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 191/386] 10.222.0.33 () {56 vars in 1114 bytes} [Sun Oct 9 00:42:39 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 230 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:42:40.206124 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd50dbc5-e955-4f33-8d02-74d8cda5fbf2 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:40.206581 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd50dbc5-e955-4f33-8d02-74d8cda5fbf2 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:40.212097 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd50dbc5-e955-4f33-8d02-74d8cda5fbf2 None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:40.212097 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd50dbc5-e955-4f33-8d02-74d8cda5fbf2 None admin] RBAC: Adding query filter params (name=member) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:40.212097 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd50dbc5-e955-4f33-8d02-74d8cda5fbf2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:40.212097 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd50dbc5-e955-4f33-8d02-74d8cda5fbf2 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:40.214772 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c71e9ad-6203-4f7a-919f-89b5342b623f None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:40.225175 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 196/387] 10.222.0.33 () {56 vars in 1101 bytes} [Sun Oct 9 00:42:39 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 251 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:42:40.241160 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd50dbc5-e955-4f33-8d02-74d8cda5fbf2 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:40.318840 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:40.326860 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c71e9ad-6203-4f7a-919f-89b5342b623f None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:40.362786 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e4d12d-a1b6-451e-810c-d70b5a5fff43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:40.363422 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e4d12d-a1b6-451e-810c-d70b5a5fff43 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:40.363703 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e4d12d-a1b6-451e-810c-d70b5a5fff43 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:40.364151 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e4d12d-a1b6-451e-810c-d70b5a5fff43 None admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:40.364944 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66e4d12d-a1b6-451e-810c-d70b5a5fff43 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:40.365422 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66e4d12d-a1b6-451e-810c-d70b5a5fff43 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:40.366948 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66e4d12d-a1b6-451e-810c-d70b5a5fff43 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:40.374161 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd50dbc5-e955-4f33-8d02-74d8cda5fbf2 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:40.376128 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 192/388] 10.222.0.33 () {56 vars in 1111 bytes} [Sun Oct 9 00:42:40 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:40.401710 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e4d12d-a1b6-451e-810c-d70b5a5fff43 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:40.416467 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f27fb3b5-46ad-4620-8844-e6a3aff08337 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:40.417403 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f27fb3b5-46ad-4620-8844-e6a3aff08337 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:40.417899 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f27fb3b5-46ad-4620-8844-e6a3aff08337 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:40.418366 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f27fb3b5-46ad-4620-8844-e6a3aff08337 None admin] PATH_INFO: `/v3/groups/e7758a219cfb46dd9c9560aef5255b8a` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:40.419686 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f27fb3b5-46ad-4620-8844-e6a3aff08337 None admin] RBAC: Authorizing `identity:get_group(group_id=e7758a219cfb46dd9c9560aef5255b8a)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:40.421446 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f27fb3b5-46ad-4620-8844-e6a3aff08337 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:40.423573 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 197/389] 10.222.0.33 () {56 vars in 1180 bytes} [Sun Oct 9 00:42:40 2022] GET /identity/v3/groups/e7758a219cfb46dd9c9560aef5255b8a => generated 226 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:42:40.435168 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f27fb3b5-46ad-4620-8844-e6a3aff08337 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:40.511575 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:40.590345 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e4d12d-a1b6-451e-810c-d70b5a5fff43 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:40.632812 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8263ebd-cd2b-4919-8c82-f0c231a7099f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:40.634059 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8263ebd-cd2b-4919-8c82-f0c231a7099f None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:40.636540 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8263ebd-cd2b-4919-8c82-f0c231a7099f None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:40.636835 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8263ebd-cd2b-4919-8c82-f0c231a7099f None admin] PATH_INFO: `/v3/groups/e7758a219cfb46dd9c9560aef5255b8a` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:40.640487 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f27fb3b5-46ad-4620-8844-e6a3aff08337 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:40.642382 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8263ebd-cd2b-4919-8c82-f0c231a7099f None admin] RBAC: Authorizing `identity:get_group(group_id=e7758a219cfb46dd9c9560aef5255b8a)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:40.643849 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8263ebd-cd2b-4919-8c82-f0c231a7099f None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:40.645439 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 193/390] 10.222.0.33 () {56 vars in 1180 bytes} [Sun Oct 9 00:42:40 2022] GET /identity/v3/groups/e7758a219cfb46dd9c9560aef5255b8a => generated 226 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:42:40.651315 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8263ebd-cd2b-4919-8c82-f0c231a7099f None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:40.695775 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:40.711673 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7643fe5-7799-4ceb-b322-7baa7d43ea2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:40.712651 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7643fe5-7799-4ceb-b322-7baa7d43ea2b None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:40.713139 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7643fe5-7799-4ceb-b322-7baa7d43ea2b None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:40.713568 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7643fe5-7799-4ceb-b322-7baa7d43ea2b None admin] PATH_INFO: `/v3/projects/768d646302f443f1a7d490b8c3d4bdbb` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:40.717821 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7643fe5-7799-4ceb-b322-7baa7d43ea2b None admin] RBAC: Authorizing `identity:get_project(project_id=768d646302f443f1a7d490b8c3d4bdbb)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:40.719540 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7643fe5-7799-4ceb-b322-7baa7d43ea2b None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:40.721738 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 198/391] 10.222.0.33 () {56 vars in 1186 bytes} [Sun Oct 9 00:42:40 2022] GET /identity/v3/projects/768d646302f443f1a7d490b8c3d4bdbb => generated 298 bytes in 288 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:42:40.723178 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e747892b-0edf-4d68-ae2a-8ac514d68e26 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:40.723710 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e747892b-0edf-4d68-ae2a-8ac514d68e26 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:40.724182 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e747892b-0edf-4d68-ae2a-8ac514d68e26 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:40.725900 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 199/392] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:42:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:42:40.729942 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e747892b-0edf-4d68-ae2a-8ac514d68e26 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:40.808534 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8263ebd-cd2b-4919-8c82-f0c231a7099f None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:40.837541 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:40.856305 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da7d66a7-2802-45b1-b052-5d8d65c61f95 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:40.857229 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da7d66a7-2802-45b1-b052-5d8d65c61f95 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:40.857757 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da7d66a7-2802-45b1-b052-5d8d65c61f95 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:40.858219 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da7d66a7-2802-45b1-b052-5d8d65c61f95 None admin] PATH_INFO: `/v3/projects/768d646302f443f1a7d490b8c3d4bdbb` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:40.859773 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da7d66a7-2802-45b1-b052-5d8d65c61f95 None admin] RBAC: Authorizing `identity:get_project(project_id=768d646302f443f1a7d490b8c3d4bdbb)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:40.862862 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da7d66a7-2802-45b1-b052-5d8d65c61f95 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:40.865172 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 194/393] 10.222.0.33 () {56 vars in 1186 bytes} [Sun Oct 9 00:42:40 2022] GET /identity/v3/projects/768d646302f443f1a7d490b8c3d4bdbb => generated 298 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:40.866603 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c754f2d7-4ebd-4d21-bbd9-84932c00bb9d None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:40.867086 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c754f2d7-4ebd-4d21-bbd9-84932c00bb9d None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:40.867556 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c754f2d7-4ebd-4d21-bbd9-84932c00bb9d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:40.874995 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e747892b-0edf-4d68-ae2a-8ac514d68e26 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:40.890822 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-c754f2d7-4ebd-4d21-bbd9-84932c00bb9d None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:42:40.892562 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-c754f2d7-4ebd-4d21-bbd9-84932c00bb9d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:42:40.921798 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34bcee58-90e5-4287-8338-324ed8617733 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:40.922884 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34bcee58-90e5-4287-8338-324ed8617733 None admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:40.923365 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34bcee58-90e5-4287-8338-324ed8617733 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:40.924627 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34bcee58-90e5-4287-8338-324ed8617733 None admin] PATH_INFO: `/v3/projects/768d646302f443f1a7d490b8c3d4bdbb/groups/e7758a219cfb46dd9c9560aef5255b8a/roles/588016933bab436d8b7c3ac76827356d` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:40.926848 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34bcee58-90e5-4287-8338-324ed8617733 None admin] RBAC: Authorizing `identity:create_grant(project_id=768d646302f443f1a7d490b8c3d4bdbb, group_id=e7758a219cfb46dd9c9560aef5255b8a, role_id=588016933bab436d8b7c3ac76827356d)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:40.928913 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34bcee58-90e5-4287-8338-324ed8617733 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:40.946664 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 200/394] 10.222.0.33 () {58 vars in 1442 bytes} [Sun Oct 9 00:42:40 2022] PUT /identity/v3/projects/768d646302f443f1a7d490b8c3d4bdbb/groups/e7758a219cfb46dd9c9560aef5255b8a/roles/588016933bab436d8b7c3ac76827356d => generated 0 bytes in 217 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:42:40.949304 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34bcee58-90e5-4287-8338-324ed8617733 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:40.977273 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 195/395] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:42:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 09 00:42:41.020957 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:41.076357 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34bcee58-90e5-4287-8338-324ed8617733 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:41.119243 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-148afe84-98f4-40e9-a3a7-2c3dc8f049c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:41.120012 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148afe84-98f4-40e9-a3a7-2c3dc8f049c8 None admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:41.120300 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148afe84-98f4-40e9-a3a7-2c3dc8f049c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:41.120583 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148afe84-98f4-40e9-a3a7-2c3dc8f049c8 None admin] PATH_INFO: `/v3/projects/768d646302f443f1a7d490b8c3d4bdbb/groups/e7758a219cfb46dd9c9560aef5255b8a/roles/7c1fdac272a34be99661077b0f1b8014` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:41.122850 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-148afe84-98f4-40e9-a3a7-2c3dc8f049c8 None admin] RBAC: Authorizing `identity:create_grant(project_id=768d646302f443f1a7d490b8c3d4bdbb, group_id=e7758a219cfb46dd9c9560aef5255b8a, role_id=7c1fdac272a34be99661077b0f1b8014)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:41.124377 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-148afe84-98f4-40e9-a3a7-2c3dc8f049c8 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:41.148564 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 201/396] 10.222.0.33 () {58 vars in 1442 bytes} [Sun Oct 9 00:42:40 2022] PUT /identity/v3/projects/768d646302f443f1a7d490b8c3d4bdbb/groups/e7758a219cfb46dd9c9560aef5255b8a/roles/7c1fdac272a34be99661077b0f1b8014 => generated 0 bytes in 200 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:42:41.347547 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c754f2d7-4ebd-4d21-bbd9-84932c00bb9d None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:41.477651 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c754f2d7-4ebd-4d21-bbd9-84932c00bb9d None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:41.501799 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:41.518142 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-731f07ea-9047-478f-ad70-56e51e5f67f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:41.518791 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-731f07ea-9047-478f-ad70-56e51e5f67f5 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:41.519071 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-731f07ea-9047-478f-ad70-56e51e5f67f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:41.519342 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-731f07ea-9047-478f-ad70-56e51e5f67f5 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:41.529992 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-731f07ea-9047-478f-ad70-56e51e5f67f5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:42:41.535009 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-731f07ea-9047-478f-ad70-56e51e5f67f5 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 09 00:42:41.535009 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:42:41.535009 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:42:41.535009 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:42:41.535009 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:42:41.535009 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:42:41.535009 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:42:41.535009 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:42:41.535009 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:42:41.535009 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:42:41.535009 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:42:41.535009 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:42:41.535009 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:42:41.535009 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:42:41.535009 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:42:41.535009 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:42:41.535989 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:42:41.535989 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:42:41.535989 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:42:41.535989 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:42:41.535989 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:42:41.535989 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:42:41.535989 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:42:41.535989 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:42:41.535989 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:42:41.535989 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:42:41.535989 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:42:41.535989 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:42:41.535989 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:42:41.535989 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:42:41.535989 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:42:41.535989 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:42:41.536764 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:42:41.536764 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:42:41.536764 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:42:41.536764 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:42:41.536764 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:42:41.536764 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:42:41.536764 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:42:41.536764 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:42:41.536764 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:42:41.536764 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:42:41.536764 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 09 00:42:41.536764 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:42:41.537809 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 196/397] 10.222.0.33 () {56 vars in 1099 bytes} [Sun Oct 9 00:42:41 2022] GET /identity/v3/roles/member => generated 84 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:42:41.542162 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-148afe84-98f4-40e9-a3a7-2c3dc8f049c8 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:41.693046 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-148afe84-98f4-40e9-a3a7-2c3dc8f049c8 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:41.709959 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:41.730660 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82240b49-257d-48bf-a620-43c4e39fdddd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:41.731272 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82240b49-257d-48bf-a620-43c4e39fdddd None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:41.731555 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82240b49-257d-48bf-a620-43c4e39fdddd None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:41.731853 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82240b49-257d-48bf-a620-43c4e39fdddd None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:41.732513 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82240b49-257d-48bf-a620-43c4e39fdddd None admin] RBAC: Adding query filter params (name=member) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:41.732812 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82240b49-257d-48bf-a620-43c4e39fdddd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:41.734277 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82240b49-257d-48bf-a620-43c4e39fdddd None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:41.750654 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 202/398] 10.222.0.33 () {56 vars in 1101 bytes} [Sun Oct 9 00:42:41 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:41.756416 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-731f07ea-9047-478f-ad70-56e51e5f67f5 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:41.893125 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-731f07ea-9047-478f-ad70-56e51e5f67f5 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:41.934431 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3474ccb-5a64-4871-adb2-63d58177b8e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:41.935068 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3474ccb-5a64-4871-adb2-63d58177b8e0 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:41.935351 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3474ccb-5a64-4871-adb2-63d58177b8e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:41.935653 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3474ccb-5a64-4871-adb2-63d58177b8e0 None admin] PATH_INFO: `/v3/groups/e7758a219cfb46dd9c9560aef5255b8a` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:41.936785 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3474ccb-5a64-4871-adb2-63d58177b8e0 None admin] RBAC: Authorizing `identity:get_group(group_id=e7758a219cfb46dd9c9560aef5255b8a)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:41.938258 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3474ccb-5a64-4871-adb2-63d58177b8e0 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:41.940236 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 197/399] 10.222.0.33 () {56 vars in 1180 bytes} [Sun Oct 9 00:42:41 2022] GET /identity/v3/groups/e7758a219cfb46dd9c9560aef5255b8a => generated 226 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:41.945526 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82240b49-257d-48bf-a620-43c4e39fdddd None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:42.086168 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82240b49-257d-48bf-a620-43c4e39fdddd None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:42.126968 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e219b4a8-8efd-4d30-a728-2f3c164b75b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:42.127616 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e219b4a8-8efd-4d30-a728-2f3c164b75b2 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:42.127900 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e219b4a8-8efd-4d30-a728-2f3c164b75b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:42.128169 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e219b4a8-8efd-4d30-a728-2f3c164b75b2 None admin] PATH_INFO: `/v3/projects/dd8011750cb34075a6f8106533f848c2` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:42.129293 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e219b4a8-8efd-4d30-a728-2f3c164b75b2 None admin] RBAC: Authorizing `identity:get_project(project_id=dd8011750cb34075a6f8106533f848c2)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:42.130735 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e219b4a8-8efd-4d30-a728-2f3c164b75b2 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:42.132663 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 203/400] 10.222.0.33 () {56 vars in 1186 bytes} [Sun Oct 9 00:42:41 2022] GET /identity/v3/projects/dd8011750cb34075a6f8106533f848c2 => generated 302 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:42.138628 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3474ccb-5a64-4871-adb2-63d58177b8e0 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:42.162267 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbd74c24-efbd-4db0-b980-1e2f162eb9c5 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:42.162479 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbd74c24-efbd-4db0-b980-1e2f162eb9c5 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:42.162701 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbd74c24-efbd-4db0-b980-1e2f162eb9c5 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:42.164588 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 204/401] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:42:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:42:42.179749 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16614f22-dc1d-4728-a950-077d1cf84bf1 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:42.180595 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16614f22-dc1d-4728-a950-077d1cf84bf1 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:42.180841 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16614f22-dc1d-4728-a950-077d1cf84bf1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:42.221236 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-16614f22-dc1d-4728-a950-077d1cf84bf1 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:42:42.223081 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-16614f22-dc1d-4728-a950-077d1cf84bf1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:42:42.223647 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:42.313271 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:42.322545 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3474ccb-5a64-4871-adb2-63d58177b8e0 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:42.342931 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 205/402] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:42:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 09 00:42:42.344508 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ee63af-54ad-4555-b357-6cd093dac537 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:42.344938 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ee63af-54ad-4555-b357-6cd093dac537 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:42.345357 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ee63af-54ad-4555-b357-6cd093dac537 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:42.346952 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 206/403] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:42:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:42:42.352604 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b023c9-d0b5-48dc-ab5d-4fc2b3658d9c None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:42.352890 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b023c9-d0b5-48dc-ab5d-4fc2b3658d9c None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:42.353162 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b023c9-d0b5-48dc-ab5d-4fc2b3658d9c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:42.369945 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aeba7a1-3bbf-47df-b11e-f52112485d0f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:42.371306 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aeba7a1-3bbf-47df-b11e-f52112485d0f None admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:42.372322 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aeba7a1-3bbf-47df-b11e-f52112485d0f None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:42.372783 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aeba7a1-3bbf-47df-b11e-f52112485d0f None admin] PATH_INFO: `/v3/projects/dd8011750cb34075a6f8106533f848c2/groups/e7758a219cfb46dd9c9560aef5255b8a/roles/588016933bab436d8b7c3ac76827356d` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:42.375180 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aeba7a1-3bbf-47df-b11e-f52112485d0f None admin] RBAC: Authorizing `identity:create_grant(project_id=dd8011750cb34075a6f8106533f848c2, group_id=e7758a219cfb46dd9c9560aef5255b8a, role_id=588016933bab436d8b7c3ac76827356d)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:42.376909 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aeba7a1-3bbf-47df-b11e-f52112485d0f None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:42.378066 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-b0b023c9-d0b5-48dc-ab5d-4fc2b3658d9c None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:42:42.379323 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-b0b023c9-d0b5-48dc-ab5d-4fc2b3658d9c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:42:42.407030 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 198/404] 10.222.0.33 () {58 vars in 1442 bytes} [Sun Oct 9 00:42:42 2022] PUT /identity/v3/projects/dd8011750cb34075a6f8106533f848c2/groups/e7758a219cfb46dd9c9560aef5255b8a/roles/588016933bab436d8b7c3ac76827356d => generated 0 bytes in 269 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:42:42.465527 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 207/405] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:42:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 09 00:42:42.724815 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aeba7a1-3bbf-47df-b11e-f52112485d0f None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:42.745426 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0b023c9-d0b5-48dc-ab5d-4fc2b3658d9c None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:42.853593 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aeba7a1-3bbf-47df-b11e-f52112485d0f None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:42.865346 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:42.888221 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0b023c9-d0b5-48dc-ab5d-4fc2b3658d9c None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:42.892062 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02526601-cbdb-47de-834e-fedff8e5fc28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:42.892701 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02526601-cbdb-47de-834e-fedff8e5fc28 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:42.892979 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02526601-cbdb-47de-834e-fedff8e5fc28 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:42.893247 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02526601-cbdb-47de-834e-fedff8e5fc28 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:42.896271 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:42.906040 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02526601-cbdb-47de-834e-fedff8e5fc28 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:42:42.911564 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-02526601-cbdb-47de-834e-fedff8e5fc28 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 09 00:42:42.911564 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:42:42.911564 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:42:42.911564 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:42:42.911564 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:42:42.911564 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:42:42.911564 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:42:42.911564 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:42:42.911564 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:42:42.911564 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:42:42.911564 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:42:42.911564 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:42:42.911564 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:42:42.911564 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:42:42.911564 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:42:42.911564 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:42:42.912622 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:42:42.912622 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:42:42.912622 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:42:42.912622 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:42:42.912622 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:42:42.912622 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:42:42.912622 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:42:42.912622 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:42:42.912622 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:42:42.912622 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:42:42.912622 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:42:42.912622 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:42:42.912622 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:42:42.912622 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:42:42.912622 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:42:42.912622 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:42:42.913844 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:42:42.913844 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:42:42.913844 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:42:42.913844 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:42:42.913844 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:42:42.913844 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:42:42.913844 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:42:42.913844 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:42:42.913844 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:42:42.913844 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:42:42.913844 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 09 00:42:42.913844 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:42:42.916649 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 199/406] 10.222.0.33 () {56 vars in 1114 bytes} [Sun Oct 9 00:42:42 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:42:42.922019 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02526601-cbdb-47de-834e-fedff8e5fc28 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:42.949820 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd92f176-c330-4ca3-9b2f-5f6c70d59842 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:42.950765 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd92f176-c330-4ca3-9b2f-5f6c70d59842 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:42.951187 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd92f176-c330-4ca3-9b2f-5f6c70d59842 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:42.951639 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd92f176-c330-4ca3-9b2f-5f6c70d59842 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:42.973294 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd92f176-c330-4ca3-9b2f-5f6c70d59842 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:42:42.983789 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-cd92f176-c330-4ca3-9b2f-5f6c70d59842 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 09 00:42:42.983789 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:42:42.983789 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:42:42.983789 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:42:42.983789 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:42:42.983789 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:42:42.983789 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:42:42.983789 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:42:42.983789 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:42:42.983789 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:42:42.983789 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:42:42.983789 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:42:42.983789 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:42:42.983789 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:42:42.983789 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:42:42.983789 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:42:42.984882 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:42:42.984882 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:42:42.984882 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:42:42.984882 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:42:42.984882 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:42:42.984882 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:42:42.984882 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:42:42.984882 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:42:42.984882 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:42:42.984882 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:42:42.984882 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:42:42.984882 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:42:42.984882 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:42:42.984882 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:42:42.984882 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:42:42.984882 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:42:42.985837 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:42:42.985837 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:42:42.985837 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:42:42.985837 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:42:42.985837 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:42:42.985837 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:42:42.985837 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:42:42.985837 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:42:42.985837 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:42:42.985837 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:42:42.985837 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 09 00:42:42.985837 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:42:42.990356 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 208/407] 10.222.0.33 () {56 vars in 1096 bytes} [Sun Oct 9 00:42:42 2022] GET /identity/v3/roles/admin => generated 83 bytes in 246 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:42:42.996003 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd92f176-c330-4ca3-9b2f-5f6c70d59842 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:43.136217 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02526601-cbdb-47de-834e-fedff8e5fc28 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:43.178519 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13c182dc-6ee3-48a4-8485-53008eb49310 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:43.179128 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13c182dc-6ee3-48a4-8485-53008eb49310 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:43.179409 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13c182dc-6ee3-48a4-8485-53008eb49310 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:43.179713 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13c182dc-6ee3-48a4-8485-53008eb49310 None admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:43.180947 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13c182dc-6ee3-48a4-8485-53008eb49310 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:43.181252 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13c182dc-6ee3-48a4-8485-53008eb49310 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:43.182741 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13c182dc-6ee3-48a4-8485-53008eb49310 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:43.188765 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd92f176-c330-4ca3-9b2f-5f6c70d59842 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:43.194362 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 200/408] 10.222.0.33 () {56 vars in 1111 bytes} [Sun Oct 9 00:42:42 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 273 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:43.201047 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13c182dc-6ee3-48a4-8485-53008eb49310 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:43.257620 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af7d8ddc-9cd9-4e97-a32a-6469b27ae200 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:43.258183 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af7d8ddc-9cd9-4e97-a32a-6469b27ae200 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:43.258475 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af7d8ddc-9cd9-4e97-a32a-6469b27ae200 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:43.258748 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af7d8ddc-9cd9-4e97-a32a-6469b27ae200 None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:43.259408 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af7d8ddc-9cd9-4e97-a32a-6469b27ae200 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:43.259734 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af7d8ddc-9cd9-4e97-a32a-6469b27ae200 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:43.263960 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:43.264329 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af7d8ddc-9cd9-4e97-a32a-6469b27ae200 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:43.277815 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 209/409] 10.222.0.33 () {56 vars in 1099 bytes} [Sun Oct 9 00:42:42 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 282 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:43.283383 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af7d8ddc-9cd9-4e97-a32a-6469b27ae200 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:43.321951 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:43.392530 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13c182dc-6ee3-48a4-8485-53008eb49310 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:43.413943 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af7d8ddc-9cd9-4e97-a32a-6469b27ae200 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:43.453161 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad9ab4d7-a2f7-413b-95e6-1c81acde1ac7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:43.453946 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9ab4d7-a2f7-413b-95e6-1c81acde1ac7 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:43.454231 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9ab4d7-a2f7-413b-95e6-1c81acde1ac7 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:43.454504 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9ab4d7-a2f7-413b-95e6-1c81acde1ac7 None admin] PATH_INFO: `/v3/groups/3026808ce79d453990adc3fe4e692084` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:43.455715 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad9ab4d7-a2f7-413b-95e6-1c81acde1ac7 None admin] RBAC: Authorizing `identity:get_group(group_id=3026808ce79d453990adc3fe4e692084)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:43.457209 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad9ab4d7-a2f7-413b-95e6-1c81acde1ac7 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:43.459069 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 210/410] 10.222.0.33 () {56 vars in 1180 bytes} [Sun Oct 9 00:42:43 2022] GET /identity/v3/groups/3026808ce79d453990adc3fe4e692084 => generated 229 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:43.464620 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad9ab4d7-a2f7-413b-95e6-1c81acde1ac7 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:43.466799 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfdd5aa1-ac39-409c-ab93-88c82ede015b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:43.467708 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfdd5aa1-ac39-409c-ab93-88c82ede015b None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:43.468214 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfdd5aa1-ac39-409c-ab93-88c82ede015b None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:43.468678 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfdd5aa1-ac39-409c-ab93-88c82ede015b None admin] PATH_INFO: `/v3/groups/e7758a219cfb46dd9c9560aef5255b8a` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:43.469815 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfdd5aa1-ac39-409c-ab93-88c82ede015b None admin] RBAC: Authorizing `identity:get_group(group_id=e7758a219cfb46dd9c9560aef5255b8a)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:43.471258 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfdd5aa1-ac39-409c-ab93-88c82ede015b None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:43.473142 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 201/411] 10.222.0.33 () {56 vars in 1180 bytes} [Sun Oct 9 00:42:43 2022] GET /identity/v3/groups/e7758a219cfb46dd9c9560aef5255b8a => generated 226 bytes in 272 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:43.480627 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfdd5aa1-ac39-409c-ab93-88c82ede015b None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:43.642232 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfdd5aa1-ac39-409c-ab93-88c82ede015b None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:43.660683 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:43.668971 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:43.682307 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a47c303-f35d-48ee-9582-cbc5afa7b4af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:43.683278 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a47c303-f35d-48ee-9582-cbc5afa7b4af None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:43.683788 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a47c303-f35d-48ee-9582-cbc5afa7b4af None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:43.684291 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a47c303-f35d-48ee-9582-cbc5afa7b4af None admin] PATH_INFO: `/v3/projects/dd8011750cb34075a6f8106533f848c2` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:43.685645 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a47c303-f35d-48ee-9582-cbc5afa7b4af None admin] RBAC: Authorizing `identity:get_project(project_id=dd8011750cb34075a6f8106533f848c2)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:43.687289 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a47c303-f35d-48ee-9582-cbc5afa7b4af None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:43.689655 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 202/412] 10.222.0.33 () {56 vars in 1186 bytes} [Sun Oct 9 00:42:43 2022] GET /identity/v3/projects/dd8011750cb34075a6f8106533f848c2 => generated 302 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:43.691528 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad9ab4d7-a2f7-413b-95e6-1c81acde1ac7 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:43.701073 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a47c303-f35d-48ee-9582-cbc5afa7b4af None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:43.744324 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07317566-15e3-48a9-b1cb-aae101aaf424 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:43.745017 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07317566-15e3-48a9-b1cb-aae101aaf424 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:43.745240 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07317566-15e3-48a9-b1cb-aae101aaf424 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:43.745547 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07317566-15e3-48a9-b1cb-aae101aaf424 None admin] PATH_INFO: `/v3/projects/34e0fbee16724df3abcac7131a4822aa` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:43.746760 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07317566-15e3-48a9-b1cb-aae101aaf424 None admin] RBAC: Authorizing `identity:get_project(project_id=34e0fbee16724df3abcac7131a4822aa)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:43.748508 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07317566-15e3-48a9-b1cb-aae101aaf424 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:43.750261 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 211/413] 10.222.0.33 () {56 vars in 1186 bytes} [Sun Oct 9 00:42:43 2022] GET /identity/v3/projects/34e0fbee16724df3abcac7131a4822aa => generated 344 bytes in 286 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:42:43.755480 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07317566-15e3-48a9-b1cb-aae101aaf424 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:43.835638 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a47c303-f35d-48ee-9582-cbc5afa7b4af None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:43.873691 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a48ea0f-7e6a-4e72-913d-8676cc9a5858 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:43.874752 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a48ea0f-7e6a-4e72-913d-8676cc9a5858 None admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:43.875076 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07317566-15e3-48a9-b1cb-aae101aaf424 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:43.875467 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a48ea0f-7e6a-4e72-913d-8676cc9a5858 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:43.875986 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a48ea0f-7e6a-4e72-913d-8676cc9a5858 None admin] PATH_INFO: `/v3/projects/dd8011750cb34075a6f8106533f848c2/groups/e7758a219cfb46dd9c9560aef5255b8a/roles/7c1fdac272a34be99661077b0f1b8014` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:43.878648 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a48ea0f-7e6a-4e72-913d-8676cc9a5858 None admin] RBAC: Authorizing `identity:create_grant(project_id=dd8011750cb34075a6f8106533f848c2, group_id=e7758a219cfb46dd9c9560aef5255b8a, role_id=7c1fdac272a34be99661077b0f1b8014)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:43.880353 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a48ea0f-7e6a-4e72-913d-8676cc9a5858 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:43.897401 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 203/414] 10.222.0.33 () {58 vars in 1442 bytes} [Sun Oct 9 00:42:43 2022] PUT /identity/v3/projects/dd8011750cb34075a6f8106533f848c2/groups/e7758a219cfb46dd9c9560aef5255b8a/roles/7c1fdac272a34be99661077b0f1b8014 => generated 0 bytes in 196 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:42:43.899725 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da5ad16-0508-49a1-b575-0913ce328417 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:43.900259 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da5ad16-0508-49a1-b575-0913ce328417 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:43.900727 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da5ad16-0508-49a1-b575-0913ce328417 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:43.906999 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 204/415] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:42:43 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:42:43.908912 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-125df240-b89b-479f-892f-0978d9e964b8 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:43.909466 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-125df240-b89b-479f-892f-0978d9e964b8 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:43.910003 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-125df240-b89b-479f-892f-0978d9e964b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:43.932482 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5beb943-d29f-490b-8aac-2fa75dd69e60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:43.933004 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-125df240-b89b-479f-892f-0978d9e964b8 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:42:43.933184 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5beb943-d29f-490b-8aac-2fa75dd69e60 None admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:43.933449 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5beb943-d29f-490b-8aac-2fa75dd69e60 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:43.933732 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5beb943-d29f-490b-8aac-2fa75dd69e60 None admin] PATH_INFO: `/v3/projects/34e0fbee16724df3abcac7131a4822aa/groups/3026808ce79d453990adc3fe4e692084/roles/092036ee46ac427baf4d56468575f697` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:43.935064 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-125df240-b89b-479f-892f-0978d9e964b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:42:43.935801 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5beb943-d29f-490b-8aac-2fa75dd69e60 None admin] RBAC: Authorizing `identity:create_grant(project_id=34e0fbee16724df3abcac7131a4822aa, group_id=3026808ce79d453990adc3fe4e692084, role_id=092036ee46ac427baf4d56468575f697)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:43.938337 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5beb943-d29f-490b-8aac-2fa75dd69e60 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:43.963098 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:43.976293 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 212/416] 10.222.0.33 () {58 vars in 1442 bytes} [Sun Oct 9 00:42:43 2022] PUT /identity/v3/projects/34e0fbee16724df3abcac7131a4822aa/groups/3026808ce79d453990adc3fe4e692084/roles/092036ee46ac427baf4d56468575f697 => generated 0 bytes in 221 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 09 00:42:44.020837 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 205/417] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:42:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 09 00:42:44.165031 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8742caed-0fb2-4652-9c6b-66efa1f35915 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:44.165357 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8742caed-0fb2-4652-9c6b-66efa1f35915 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:44.165577 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8742caed-0fb2-4652-9c6b-66efa1f35915 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:44.167073 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 213/418] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:42:44 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:42:44.172958 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b102249-2a8c-4545-8d03-9b66d77e35e8 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:44.173556 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b102249-2a8c-4545-8d03-9b66d77e35e8 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:44.174048 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b102249-2a8c-4545-8d03-9b66d77e35e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:44.195407 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-3b102249-2a8c-4545-8d03-9b66d77e35e8 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:42:44.198455 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-3b102249-2a8c-4545-8d03-9b66d77e35e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:42:44.273524 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 206/419] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:42:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 09 00:42:44.326447 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8742caed-0fb2-4652-9c6b-66efa1f35915 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:44.354792 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:44.459518 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8742caed-0fb2-4652-9c6b-66efa1f35915 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:44.507251 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00be0159-bf77-4726-a1a2-70809e6b2a05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:44.508283 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00be0159-bf77-4726-a1a2-70809e6b2a05 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:44.508763 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00be0159-bf77-4726-a1a2-70809e6b2a05 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:44.509226 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00be0159-bf77-4726-a1a2-70809e6b2a05 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:44.523130 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00be0159-bf77-4726-a1a2-70809e6b2a05 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:42:44.529225 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-00be0159-bf77-4726-a1a2-70809e6b2a05 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 09 00:42:44.529225 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:42:44.529225 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:42:44.529225 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:42:44.529225 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:42:44.529225 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:42:44.529225 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:42:44.529225 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:42:44.529225 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:42:44.529225 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:42:44.529225 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:42:44.529225 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:42:44.529225 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:42:44.529225 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:42:44.529225 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:42:44.529225 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:42:44.530074 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:42:44.530074 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:42:44.530074 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:42:44.530074 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:42:44.530074 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:42:44.530074 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:42:44.530074 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:42:44.530074 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:42:44.530074 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:42:44.530074 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:42:44.530074 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:42:44.530074 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:42:44.530074 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:42:44.530074 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:42:44.530074 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:42:44.530074 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:42:44.530874 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:42:44.530874 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:42:44.530874 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:42:44.530874 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:42:44.530874 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:42:44.530874 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:42:44.530874 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:42:44.530874 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:42:44.530874 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:42:44.530874 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:42:44.530874 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 09 00:42:44.530874 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:42:44.533149 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 214/420] 10.222.0.33 () {56 vars in 1114 bytes} [Sun Oct 9 00:42:44 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:42:44.539332 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b102249-2a8c-4545-8d03-9b66d77e35e8 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:44.586727 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00be0159-bf77-4726-a1a2-70809e6b2a05 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:44.671126 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:44.691814 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:44.699965 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b102249-2a8c-4545-8d03-9b66d77e35e8 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:44.724550 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00be0159-bf77-4726-a1a2-70809e6b2a05 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:44.764433 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21e4264e-a9a1-4678-a23c-c9b60ecdc227 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:44.765398 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21e4264e-a9a1-4678-a23c-c9b60ecdc227 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:44.765900 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21e4264e-a9a1-4678-a23c-c9b60ecdc227 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:44.766366 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21e4264e-a9a1-4678-a23c-c9b60ecdc227 None admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:44.767231 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21e4264e-a9a1-4678-a23c-c9b60ecdc227 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:44.767755 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21e4264e-a9a1-4678-a23c-c9b60ecdc227 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:44.770613 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21e4264e-a9a1-4678-a23c-c9b60ecdc227 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:44.785921 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 207/421] 10.222.0.33 () {56 vars in 1111 bytes} [Sun Oct 9 00:42:44 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:44.787850 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7da2a18-1dca-47d1-9fd5-a05d7b875954 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:44.789992 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7da2a18-1dca-47d1-9fd5-a05d7b875954 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:44.790487 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7da2a18-1dca-47d1-9fd5-a05d7b875954 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:44.790957 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7da2a18-1dca-47d1-9fd5-a05d7b875954 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:44.792426 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21e4264e-a9a1-4678-a23c-c9b60ecdc227 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:44.812438 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7da2a18-1dca-47d1-9fd5-a05d7b875954 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:42:44.829224 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-c7da2a18-1dca-47d1-9fd5-a05d7b875954 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 09 00:42:44.829224 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:42:44.829224 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:42:44.829224 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:42:44.829224 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:42:44.829224 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:42:44.829224 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:42:44.829224 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:42:44.829224 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:42:44.829224 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:42:44.829224 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:42:44.829224 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:42:44.829224 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:42:44.829224 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:42:44.829224 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:42:44.829224 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:42:44.830131 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:42:44.830131 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:42:44.830131 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:42:44.830131 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:42:44.830131 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:42:44.830131 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:42:44.830131 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:42:44.830131 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:42:44.830131 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:42:44.830131 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:42:44.830131 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:42:44.830131 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:42:44.830131 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:42:44.830131 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:42:44.830131 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:42:44.830131 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:42:44.830899 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:42:44.830899 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:42:44.830899 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:42:44.830899 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:42:44.830899 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:42:44.830899 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:42:44.830899 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:42:44.830899 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:42:44.830899 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:42:44.830899 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:42:44.830899 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 09 00:42:44.830899 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:42:44.833208 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 215/422] 10.222.0.33 () {56 vars in 1099 bytes} [Sun Oct 9 00:42:44 2022] GET /identity/v3/roles/member => generated 84 bytes in 247 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:42:44.839000 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7da2a18-1dca-47d1-9fd5-a05d7b875954 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:44.968859 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21e4264e-a9a1-4678-a23c-c9b60ecdc227 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:44.969634 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7da2a18-1dca-47d1-9fd5-a05d7b875954 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:45.009702 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea36516f-d4ed-42b3-9c0d-0402bc7bb035 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:45.010629 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea36516f-d4ed-42b3-9c0d-0402bc7bb035 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:45.011141 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea36516f-d4ed-42b3-9c0d-0402bc7bb035 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:45.011675 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea36516f-d4ed-42b3-9c0d-0402bc7bb035 None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:45.012532 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea36516f-d4ed-42b3-9c0d-0402bc7bb035 None admin] RBAC: Adding query filter params (name=member) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:45.013287 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea36516f-d4ed-42b3-9c0d-0402bc7bb035 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:45.015121 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea36516f-d4ed-42b3-9c0d-0402bc7bb035 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:45.027883 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 216/423] 10.222.0.33 () {56 vars in 1101 bytes} [Sun Oct 9 00:42:44 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:42:45.033482 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea36516f-d4ed-42b3-9c0d-0402bc7bb035 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:45.040170 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05057978-53c3-43ac-92bb-ee469595407d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:45.041054 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05057978-53c3-43ac-92bb-ee469595407d None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:45.041899 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05057978-53c3-43ac-92bb-ee469595407d None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:45.044973 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05057978-53c3-43ac-92bb-ee469595407d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:45.051959 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:45.066781 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:45.123497 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05057978-53c3-43ac-92bb-ee469595407d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:45.125427 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05057978-53c3-43ac-92bb-ee469595407d None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:45.232164 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 208/424] 10.222.0.33 () {58 vars in 1282 bytes} [Sun Oct 9 00:42:44 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 440 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 09 00:42:45.239208 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05057978-53c3-43ac-92bb-ee469595407d None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:45.303498 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea36516f-d4ed-42b3-9c0d-0402bc7bb035 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:45.313570 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:45.353030 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f681d6f7-1921-4183-8ac4-6f3f9f4ca557 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:45.353601 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f681d6f7-1921-4183-8ac4-6f3f9f4ca557 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:45.353880 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f681d6f7-1921-4183-8ac4-6f3f9f4ca557 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:45.354147 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f681d6f7-1921-4183-8ac4-6f3f9f4ca557 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:45.370328 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05057978-53c3-43ac-92bb-ee469595407d None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:45.408600 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4664a5ec-ac99-4e1e-bde5-c8f67d302c6d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:45.409583 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4664a5ec-ac99-4e1e-bde5-c8f67d302c6d None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:45.410076 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4664a5ec-ac99-4e1e-bde5-c8f67d302c6d None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:45.410540 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4664a5ec-ac99-4e1e-bde5-c8f67d302c6d None admin] PATH_INFO: `/v3/projects/768d646302f443f1a7d490b8c3d4bdbb` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:45.411883 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4664a5ec-ac99-4e1e-bde5-c8f67d302c6d None admin] RBAC: Authorizing `identity:get_project(project_id=768d646302f443f1a7d490b8c3d4bdbb)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:45.413766 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4664a5ec-ac99-4e1e-bde5-c8f67d302c6d None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:45.416595 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 209/425] 10.222.0.33 () {56 vars in 1186 bytes} [Sun Oct 9 00:42:45 2022] GET /identity/v3/projects/768d646302f443f1a7d490b8c3d4bdbb => generated 298 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:42:45.418091 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-447ce09b-59e5-4c03-b983-23bae53a95a1 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:45.418615 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-447ce09b-59e5-4c03-b983-23bae53a95a1 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:45.419099 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-447ce09b-59e5-4c03-b983-23bae53a95a1 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:45.420777 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 210/426] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:42:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:42:45.422389 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-447ce09b-59e5-4c03-b983-23bae53a95a1 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:45.456673 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f681d6f7-1921-4183-8ac4-6f3f9f4ca557 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:45.458442 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f681d6f7-1921-4183-8ac4-6f3f9f4ca557 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:45.466827 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:45.530564 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-447ce09b-59e5-4c03-b983-23bae53a95a1 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:45.548214 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:45.567171 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67077745-e22b-4a89-8fc2-d902037a89a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:45.567762 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67077745-e22b-4a89-8fc2-d902037a89a0 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:45.568045 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67077745-e22b-4a89-8fc2-d902037a89a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:45.568318 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67077745-e22b-4a89-8fc2-d902037a89a0 None admin] PATH_INFO: `/v3/groups/e7758a219cfb46dd9c9560aef5255b8a` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:45.569426 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67077745-e22b-4a89-8fc2-d902037a89a0 None admin] RBAC: Authorizing `identity:get_group(group_id=e7758a219cfb46dd9c9560aef5255b8a)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:45.569714 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 217/427] 10.222.0.33 () {58 vars in 1282 bytes} [Sun Oct 9 00:42:45 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 532 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 09 00:42:45.571879 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67077745-e22b-4a89-8fc2-d902037a89a0 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:45.573014 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff1ae6e-e72b-4124-956a-ca798209f09d None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:45.573296 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff1ae6e-e72b-4124-956a-ca798209f09d None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:45.573568 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff1ae6e-e72b-4124-956a-ca798209f09d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:45.573809 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 211/428] 10.222.0.33 () {56 vars in 1180 bytes} [Sun Oct 9 00:42:45 2022] GET /identity/v3/groups/e7758a219cfb46dd9c9560aef5255b8a => generated 226 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:45.574576 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67077745-e22b-4a89-8fc2-d902037a89a0 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:45.597109 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-4ff1ae6e-e72b-4124-956a-ca798209f09d None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:42:45.598470 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-4ff1ae6e-e72b-4124-956a-ca798209f09d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:42:45.668509 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 218/429] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:42:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 09 00:42:45.669346 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ff1ae6e-e72b-4124-956a-ca798209f09d None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:45.677430 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67077745-e22b-4a89-8fc2-d902037a89a0 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:45.706484 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:45.713046 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff12af7f-3e5d-491b-8724-332eb9cc53c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:45.713684 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff12af7f-3e5d-491b-8724-332eb9cc53c9 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:45.713968 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff12af7f-3e5d-491b-8724-332eb9cc53c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:45.714241 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff12af7f-3e5d-491b-8724-332eb9cc53c9 None admin] PATH_INFO: `/v3/projects/768d646302f443f1a7d490b8c3d4bdbb` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:45.715478 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff12af7f-3e5d-491b-8724-332eb9cc53c9 None admin] RBAC: Authorizing `identity:get_project(project_id=768d646302f443f1a7d490b8c3d4bdbb)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:45.716976 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff12af7f-3e5d-491b-8724-332eb9cc53c9 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:45.718949 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 212/430] 10.222.0.33 () {56 vars in 1186 bytes} [Sun Oct 9 00:42:45 2022] GET /identity/v3/projects/768d646302f443f1a7d490b8c3d4bdbb => generated 298 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:45.727465 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff12af7f-3e5d-491b-8724-332eb9cc53c9 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:45.786957 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ff1ae6e-e72b-4124-956a-ca798209f09d None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:45.804855 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:45.829163 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ced5287-6066-44d2-aa6e-431fe36f62b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:45.830358 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ced5287-6066-44d2-aa6e-431fe36f62b1 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:45.830805 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ced5287-6066-44d2-aa6e-431fe36f62b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:45.831440 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ced5287-6066-44d2-aa6e-431fe36f62b1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:45.832367 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ced5287-6066-44d2-aa6e-431fe36f62b1 None admin] RBAC: Adding query filter params (group.id=e7758a219cfb46dd9c9560aef5255b8a, role.id=7c1fdac272a34be99661077b0f1b8014, scope.project.id=768d646302f443f1a7d490b8c3d4bdbb) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:45.832857 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ced5287-6066-44d2-aa6e-431fe36f62b1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:45.834780 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ced5287-6066-44d2-aa6e-431fe36f62b1 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:45.841664 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 219/431] 10.222.0.33 () {56 vars in 1376 bytes} [Sun Oct 9 00:42:45 2022] GET /identity/v3/role_assignments?group.id=e7758a219cfb46dd9c9560aef5255b8a&scope.project.id=768d646302f443f1a7d490b8c3d4bdbb&role.id=7c1fdac272a34be99661077b0f1b8014 => generated 611 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:45.866826 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff12af7f-3e5d-491b-8724-332eb9cc53c9 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:45.934306 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d4fd204-2d19-4331-801e-2c725d45a059 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:45.935334 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d4fd204-2d19-4331-801e-2c725d45a059 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:45.935878 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d4fd204-2d19-4331-801e-2c725d45a059 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:45.936342 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d4fd204-2d19-4331-801e-2c725d45a059 None admin] PATH_INFO: `/v3/groups/e7758a219cfb46dd9c9560aef5255b8a` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:45.937719 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d4fd204-2d19-4331-801e-2c725d45a059 None admin] RBAC: Authorizing `identity:get_group(group_id=e7758a219cfb46dd9c9560aef5255b8a)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:45.939334 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d4fd204-2d19-4331-801e-2c725d45a059 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:45.941433 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 213/432] 10.222.0.33 () {56 vars in 1180 bytes} [Sun Oct 9 00:42:45 2022] GET /identity/v3/groups/e7758a219cfb46dd9c9560aef5255b8a => generated 226 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:45.947998 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ced5287-6066-44d2-aa6e-431fe36f62b1 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:46.054448 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:46.057835 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ced5287-6066-44d2-aa6e-431fe36f62b1 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:46.110247 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-548a76c5-8ebd-4e5d-81c6-a391d8d2a302 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:46.110865 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-548a76c5-8ebd-4e5d-81c6-a391d8d2a302 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:46.111139 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-548a76c5-8ebd-4e5d-81c6-a391d8d2a302 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:46.111405 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-548a76c5-8ebd-4e5d-81c6-a391d8d2a302 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:46.112170 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-548a76c5-8ebd-4e5d-81c6-a391d8d2a302 None admin] RBAC: Adding query filter params (group.id=e7758a219cfb46dd9c9560aef5255b8a, role.id=588016933bab436d8b7c3ac76827356d, scope.project.id=768d646302f443f1a7d490b8c3d4bdbb) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:46.112475 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-548a76c5-8ebd-4e5d-81c6-a391d8d2a302 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:46.113954 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-548a76c5-8ebd-4e5d-81c6-a391d8d2a302 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:46.117312 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d4fd204-2d19-4331-801e-2c725d45a059 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:46.125270 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 220/433] 10.222.0.33 () {56 vars in 1376 bytes} [Sun Oct 9 00:42:45 2022] GET /identity/v3/role_assignments?group.id=e7758a219cfb46dd9c9560aef5255b8a&scope.project.id=768d646302f443f1a7d490b8c3d4bdbb&role.id=588016933bab436d8b7c3ac76827356d => generated 611 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:46.237919 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d4fd204-2d19-4331-801e-2c725d45a059 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:46.261935 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:46.278467 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6d890c7-3c25-4029-8935-a3aeb903477b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:46.279175 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6d890c7-3c25-4029-8935-a3aeb903477b None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:46.279460 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6d890c7-3c25-4029-8935-a3aeb903477b None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:46.279771 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6d890c7-3c25-4029-8935-a3aeb903477b None admin] PATH_INFO: `/v3/roles/member` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:46.292992 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6d890c7-3c25-4029-8935-a3aeb903477b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:42:46.298587 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-b6d890c7-3c25-4029-8935-a3aeb903477b None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 09 00:42:46.298587 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:42:46.298587 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:42:46.298587 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:42:46.298587 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:42:46.298587 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:42:46.298587 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:42:46.298587 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:42:46.298587 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:42:46.298587 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:42:46.298587 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:42:46.298587 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:42:46.298587 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:42:46.298587 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:42:46.298587 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:42:46.298587 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:42:46.299506 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:42:46.299506 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:42:46.299506 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:42:46.299506 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:42:46.299506 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:42:46.299506 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:42:46.299506 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:42:46.299506 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:42:46.299506 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:42:46.299506 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:42:46.299506 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:42:46.299506 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:42:46.299506 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:42:46.299506 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:42:46.299506 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:42:46.299506 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:42:46.300311 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:42:46.300311 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:42:46.300311 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:42:46.300311 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:42:46.300311 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:42:46.300311 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:42:46.300311 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:42:46.300311 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:42:46.300311 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:42:46.300311 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:42:46.300311 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 09 00:42:46.300311 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:42:46.300311 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 214/434] 10.222.0.33 () {56 vars in 1099 bytes} [Sun Oct 9 00:42:46 2022] GET /identity/v3/roles/member => generated 84 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:42:46.306692 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-548a76c5-8ebd-4e5d-81c6-a391d8d2a302 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:46.425020 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-548a76c5-8ebd-4e5d-81c6-a391d8d2a302 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:46.467021 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f55179f-c0c3-4be2-97ea-041f53a1b2c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:46.467723 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f55179f-c0c3-4be2-97ea-041f53a1b2c8 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:46.468002 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f55179f-c0c3-4be2-97ea-041f53a1b2c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:46.468273 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f55179f-c0c3-4be2-97ea-041f53a1b2c8 None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:46.468940 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f55179f-c0c3-4be2-97ea-041f53a1b2c8 None admin] RBAC: Adding query filter params (name=member) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:46.469246 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f55179f-c0c3-4be2-97ea-041f53a1b2c8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:46.470622 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f55179f-c0c3-4be2-97ea-041f53a1b2c8 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:46.480560 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 221/435] 10.222.0.33 () {56 vars in 1101 bytes} [Sun Oct 9 00:42:46 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:46.486026 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6d890c7-3c25-4029-8935-a3aeb903477b None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:46.562166 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43555c2f-35ea-42d3-ad94-934c5c9e87f4 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:46.563063 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43555c2f-35ea-42d3-ad94-934c5c9e87f4 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:46.563538 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43555c2f-35ea-42d3-ad94-934c5c9e87f4 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:46.565230 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 222/436] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:42:46 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:42:46.572448 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f90a9f-4e93-4605-b830-0a16dd723d2f None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:46.572732 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f90a9f-4e93-4605-b830-0a16dd723d2f None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:46.574064 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f90a9f-4e93-4605-b830-0a16dd723d2f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:46.596164 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-c1f90a9f-4e93-4605-b830-0a16dd723d2f None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:42:46.597563 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-c1f90a9f-4e93-4605-b830-0a16dd723d2f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:42:46.607412 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6d890c7-3c25-4029-8935-a3aeb903477b None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:46.625995 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:46.651154 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfb0bcf8-1582-484a-a410-d7259d6688a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:46.651820 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb0bcf8-1582-484a-a410-d7259d6688a4 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:46.652264 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb0bcf8-1582-484a-a410-d7259d6688a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:46.652680 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb0bcf8-1582-484a-a410-d7259d6688a4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:46.699296 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 223/437] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:42:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 09 00:42:46.724023 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:46.750216 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfb0bcf8-1582-484a-a410-d7259d6688a4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:46.752056 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfb0bcf8-1582-484a-a410-d7259d6688a4 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:46.773531 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c331a6b0-a20b-41c1-ad55-50297d5ed637 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:46.773859 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c331a6b0-a20b-41c1-ad55-50297d5ed637 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:46.774424 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c331a6b0-a20b-41c1-ad55-50297d5ed637 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:46.776014 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 224/438] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:42:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 09 00:42:46.781325 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d83c024-cd79-4f7f-9ca1-cf5ec93b1938 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:46.781830 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d83c024-cd79-4f7f-9ca1-cf5ec93b1938 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:46.782346 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d83c024-cd79-4f7f-9ca1-cf5ec93b1938 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:46.801581 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-5d83c024-cd79-4f7f-9ca1-cf5ec93b1938 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:42:46.803092 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-5d83c024-cd79-4f7f-9ca1-cf5ec93b1938 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:42:46.877957 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 215/439] 10.222.0.33 () {58 vars in 1282 bytes} [Sun Oct 9 00:42:46 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 392 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 09 00:42:46.884080 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfb0bcf8-1582-484a-a410-d7259d6688a4 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:46.894847 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 225/440] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:42:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 09 00:42:46.971436 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d83c024-cd79-4f7f-9ca1-cf5ec93b1938 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:46.973550 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:46.993613 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfb0bcf8-1582-484a-a410-d7259d6688a4 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:47.027728 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eb10156-0f5a-4b29-a79e-428e2e13e29f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:47.028352 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb10156-0f5a-4b29-a79e-428e2e13e29f None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:47.028635 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb10156-0f5a-4b29-a79e-428e2e13e29f None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:47.028911 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb10156-0f5a-4b29-a79e-428e2e13e29f None admin] PATH_INFO: `/v3/projects/dd8011750cb34075a6f8106533f848c2` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:47.030021 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eb10156-0f5a-4b29-a79e-428e2e13e29f None admin] RBAC: Authorizing `identity:get_project(project_id=dd8011750cb34075a6f8106533f848c2)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:47.031457 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eb10156-0f5a-4b29-a79e-428e2e13e29f None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:47.033090 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 216/441] 10.222.0.33 () {56 vars in 1186 bytes} [Sun Oct 9 00:42:46 2022] GET /identity/v3/projects/dd8011750cb34075a6f8106533f848c2 => generated 302 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:42:47.038749 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eb10156-0f5a-4b29-a79e-428e2e13e29f None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:47.059516 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:47.082708 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d83c024-cd79-4f7f-9ca1-cf5ec93b1938 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:47.119933 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fefc8228-4410-445a-9da9-672f0675d62d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:47.120661 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fefc8228-4410-445a-9da9-672f0675d62d None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:47.120843 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fefc8228-4410-445a-9da9-672f0675d62d None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:47.121095 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fefc8228-4410-445a-9da9-672f0675d62d None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:47.131087 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fefc8228-4410-445a-9da9-672f0675d62d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:42:47.135900 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-fefc8228-4410-445a-9da9-672f0675d62d None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 09 00:42:47.135900 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:42:47.135900 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:42:47.135900 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:42:47.135900 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:42:47.135900 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:42:47.135900 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:42:47.135900 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:42:47.135900 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:42:47.135900 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:42:47.135900 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:42:47.135900 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:42:47.135900 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:42:47.135900 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:42:47.135900 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:42:47.135900 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:42:47.136750 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:42:47.136750 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:42:47.136750 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:42:47.136750 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:42:47.136750 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:42:47.136750 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:42:47.136750 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:42:47.136750 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:42:47.136750 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:42:47.136750 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:42:47.136750 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:42:47.136750 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:42:47.136750 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:42:47.136750 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:42:47.136750 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:42:47.136750 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:42:47.137600 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:42:47.137600 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:42:47.137600 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:42:47.137600 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:42:47.137600 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:42:47.137600 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:42:47.137600 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:42:47.137600 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:42:47.137600 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:42:47.137600 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:42:47.137600 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 09 00:42:47.137600 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:42:47.137600 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 226/442] 10.222.0.33 () {56 vars in 1114 bytes} [Sun Oct 9 00:42:46 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 09 00:42:47.142553 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fefc8228-4410-445a-9da9-672f0675d62d None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:47.148889 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eb10156-0f5a-4b29-a79e-428e2e13e29f None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:47.183349 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1953d9a-6a70-4893-b26a-428f6871afab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:47.183998 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1953d9a-6a70-4893-b26a-428f6871afab None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:47.184277 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1953d9a-6a70-4893-b26a-428f6871afab None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:47.184551 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1953d9a-6a70-4893-b26a-428f6871afab None admin] PATH_INFO: `/v3/groups/e7758a219cfb46dd9c9560aef5255b8a` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:47.185640 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1953d9a-6a70-4893-b26a-428f6871afab None admin] RBAC: Authorizing `identity:get_group(group_id=e7758a219cfb46dd9c9560aef5255b8a)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:47.187079 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1953d9a-6a70-4893-b26a-428f6871afab None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:47.188775 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 217/443] 10.222.0.33 () {56 vars in 1180 bytes} [Sun Oct 9 00:42:47 2022] GET /identity/v3/groups/e7758a219cfb46dd9c9560aef5255b8a => generated 226 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:42:47.189382 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1953d9a-6a70-4893-b26a-428f6871afab None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:47.206237 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:47.249668 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fefc8228-4410-445a-9da9-672f0675d62d None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:47.282937 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16bb372d-d755-46cc-972b-7b0f2628894a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:47.283529 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16bb372d-d755-46cc-972b-7b0f2628894a None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:47.283847 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16bb372d-d755-46cc-972b-7b0f2628894a None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:47.284116 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16bb372d-d755-46cc-972b-7b0f2628894a None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:47.284755 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16bb372d-d755-46cc-972b-7b0f2628894a None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:47.285049 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16bb372d-d755-46cc-972b-7b0f2628894a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:47.286402 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16bb372d-d755-46cc-972b-7b0f2628894a None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:47.286621 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1953d9a-6a70-4893-b26a-428f6871afab None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:47.294915 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 227/444] 10.222.0.33 () {56 vars in 1111 bytes} [Sun Oct 9 00:42:47 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:42:47.295711 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16bb372d-d755-46cc-972b-7b0f2628894a None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:47.306932 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:47.320519 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b062c47a-afa8-40b9-ab11-18f57a4a69c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:47.321140 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b062c47a-afa8-40b9-ab11-18f57a4a69c0 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:47.321411 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b062c47a-afa8-40b9-ab11-18f57a4a69c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:47.321680 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b062c47a-afa8-40b9-ab11-18f57a4a69c0 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:47.331397 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b062c47a-afa8-40b9-ab11-18f57a4a69c0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:42:47.336805 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-b062c47a-afa8-40b9-ab11-18f57a4a69c0 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 09 00:42:47.336805 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:42:47.336805 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:42:47.336805 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:42:47.336805 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:42:47.336805 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:42:47.336805 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:42:47.336805 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:42:47.336805 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:42:47.336805 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:42:47.336805 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:42:47.336805 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:42:47.336805 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:42:47.336805 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:42:47.336805 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:42:47.336805 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:42:47.337672 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:42:47.337672 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:42:47.337672 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:42:47.337672 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:42:47.337672 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:42:47.337672 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:42:47.337672 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:42:47.337672 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:42:47.337672 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:42:47.337672 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:42:47.337672 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:42:47.337672 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:42:47.337672 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:42:47.337672 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:42:47.337672 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:42:47.337672 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:42:47.338457 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:42:47.338457 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:42:47.338457 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:42:47.338457 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:42:47.338457 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:42:47.338457 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:42:47.338457 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:42:47.338457 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:42:47.338457 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:42:47.338457 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:42:47.338457 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 09 00:42:47.338457 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:42:47.338457 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 218/445] 10.222.0.33 () {56 vars in 1096 bytes} [Sun Oct 9 00:42:47 2022] GET /identity/v3/roles/admin => generated 83 bytes in 149 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:42:47.339126 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b062c47a-afa8-40b9-ab11-18f57a4a69c0 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:47.395061 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16bb372d-d755-46cc-972b-7b0f2628894a None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:47.414189 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:47.429648 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0a61ca5-3e31-499e-82ed-a96035b8210f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:47.430296 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a61ca5-3e31-499e-82ed-a96035b8210f None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:47.430573 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a61ca5-3e31-499e-82ed-a96035b8210f None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:47.430842 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a61ca5-3e31-499e-82ed-a96035b8210f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:47.431637 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0a61ca5-3e31-499e-82ed-a96035b8210f None admin] RBAC: Adding query filter params (group.id=e7758a219cfb46dd9c9560aef5255b8a, role.id=588016933bab436d8b7c3ac76827356d, scope.project.id=dd8011750cb34075a6f8106533f848c2) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:47.431950 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0a61ca5-3e31-499e-82ed-a96035b8210f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:47.433394 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0a61ca5-3e31-499e-82ed-a96035b8210f None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:47.437333 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b062c47a-afa8-40b9-ab11-18f57a4a69c0 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:47.439119 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 228/446] 10.222.0.33 () {56 vars in 1376 bytes} [Sun Oct 9 00:42:47 2022] GET /identity/v3/role_assignments?group.id=e7758a219cfb46dd9c9560aef5255b8a&scope.project.id=dd8011750cb34075a6f8106533f848c2&role.id=588016933bab436d8b7c3ac76827356d => generated 611 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:47.439914 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0a61ca5-3e31-499e-82ed-a96035b8210f None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:47.481441 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb33e7a7-e9a7-4775-bbcd-1f25001c6005 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:47.482064 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb33e7a7-e9a7-4775-bbcd-1f25001c6005 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:47.482343 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb33e7a7-e9a7-4775-bbcd-1f25001c6005 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:47.482611 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb33e7a7-e9a7-4775-bbcd-1f25001c6005 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:47.522936 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:47.557057 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0a61ca5-3e31-499e-82ed-a96035b8210f None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:47.571831 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb33e7a7-e9a7-4775-bbcd-1f25001c6005 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:47.573303 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb33e7a7-e9a7-4775-bbcd-1f25001c6005 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:47.594430 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76bddd9f-2cb1-4630-8e02-c65cd34f47a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:47.595032 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76bddd9f-2cb1-4630-8e02-c65cd34f47a4 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:47.595307 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76bddd9f-2cb1-4630-8e02-c65cd34f47a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:47.595575 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76bddd9f-2cb1-4630-8e02-c65cd34f47a4 None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:47.596274 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76bddd9f-2cb1-4630-8e02-c65cd34f47a4 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:47.596569 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76bddd9f-2cb1-4630-8e02-c65cd34f47a4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:47.597982 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76bddd9f-2cb1-4630-8e02-c65cd34f47a4 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:47.608286 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 229/447] 10.222.0.33 () {56 vars in 1099 bytes} [Sun Oct 9 00:42:47 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:47.614558 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76bddd9f-2cb1-4630-8e02-c65cd34f47a4 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:47.665544 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:47.696991 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 219/448] 10.222.0.33 () {58 vars in 1282 bytes} [Sun Oct 9 00:42:47 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 359 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 09 00:42:47.702634 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb33e7a7-e9a7-4775-bbcd-1f25001c6005 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:47.731648 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76bddd9f-2cb1-4630-8e02-c65cd34f47a4 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:47.768638 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37ba0833-937c-4278-bdd3-6bf03db941c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:47.769504 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37ba0833-937c-4278-bdd3-6bf03db941c9 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:47.769928 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37ba0833-937c-4278-bdd3-6bf03db941c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:47.770340 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37ba0833-937c-4278-bdd3-6bf03db941c9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:47.773202 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:47.834873 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb33e7a7-e9a7-4775-bbcd-1f25001c6005 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:47.849872 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37ba0833-937c-4278-bdd3-6bf03db941c9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:47.851333 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37ba0833-937c-4278-bdd3-6bf03db941c9 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:47.868233 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05000d7b-b7e4-460b-a43e-bfbda57d124f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:47.869126 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05000d7b-b7e4-460b-a43e-bfbda57d124f None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:47.869401 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05000d7b-b7e4-460b-a43e-bfbda57d124f None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:47.869670 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05000d7b-b7e4-460b-a43e-bfbda57d124f None admin] PATH_INFO: `/v3/projects/dd8011750cb34075a6f8106533f848c2` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:47.870771 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05000d7b-b7e4-460b-a43e-bfbda57d124f None admin] RBAC: Authorizing `identity:get_project(project_id=dd8011750cb34075a6f8106533f848c2)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:47.872632 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05000d7b-b7e4-460b-a43e-bfbda57d124f None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:47.874388 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 220/449] 10.222.0.33 () {56 vars in 1186 bytes} [Sun Oct 9 00:42:47 2022] GET /identity/v3/projects/dd8011750cb34075a6f8106533f848c2 => generated 302 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:42:47.879612 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05000d7b-b7e4-460b-a43e-bfbda57d124f None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:47.924417 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:47.949045 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 230/450] 10.222.0.33 () {58 vars in 1282 bytes} [Sun Oct 9 00:42:47 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 335 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 09 00:42:47.954528 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37ba0833-937c-4278-bdd3-6bf03db941c9 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:47.978297 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05000d7b-b7e4-460b-a43e-bfbda57d124f None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:47.986735 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:48.012226 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bf96c1f-701e-4464-9668-8848111972c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:48.012847 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf96c1f-701e-4464-9668-8848111972c4 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:48.013129 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf96c1f-701e-4464-9668-8848111972c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:48.013402 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf96c1f-701e-4464-9668-8848111972c4 None admin] PATH_INFO: `/v3/groups/e7758a219cfb46dd9c9560aef5255b8a` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:48.014500 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bf96c1f-701e-4464-9668-8848111972c4 None admin] RBAC: Authorizing `identity:get_group(group_id=e7758a219cfb46dd9c9560aef5255b8a)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:48.015956 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bf96c1f-701e-4464-9668-8848111972c4 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:48.017531 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 221/451] 10.222.0.33 () {56 vars in 1180 bytes} [Sun Oct 9 00:42:47 2022] GET /identity/v3/groups/e7758a219cfb46dd9c9560aef5255b8a => generated 226 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:42:48.023308 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bf96c1f-701e-4464-9668-8848111972c4 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:48.051683 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37ba0833-937c-4278-bdd3-6bf03db941c9 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:48.088299 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-295dca19-6bd2-45ba-a02d-e10cc6970aec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:48.088924 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-295dca19-6bd2-45ba-a02d-e10cc6970aec None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:48.089205 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-295dca19-6bd2-45ba-a02d-e10cc6970aec None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:48.089478 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-295dca19-6bd2-45ba-a02d-e10cc6970aec None admin] PATH_INFO: `/v3/projects/34e0fbee16724df3abcac7131a4822aa` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:48.090597 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-295dca19-6bd2-45ba-a02d-e10cc6970aec None admin] RBAC: Authorizing `identity:get_project(project_id=34e0fbee16724df3abcac7131a4822aa)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:48.092081 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-295dca19-6bd2-45ba-a02d-e10cc6970aec None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:48.093687 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 231/452] 10.222.0.33 () {56 vars in 1186 bytes} [Sun Oct 9 00:42:47 2022] GET /identity/v3/projects/34e0fbee16724df3abcac7131a4822aa => generated 344 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:42:48.099103 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-295dca19-6bd2-45ba-a02d-e10cc6970aec None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:48.121269 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bf96c1f-701e-4464-9668-8848111972c4 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:48.134718 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:48.156015 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a4950fb-f4b4-428e-9f27-222edf72bed9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:48.156630 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a4950fb-f4b4-428e-9f27-222edf72bed9 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:48.156909 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a4950fb-f4b4-428e-9f27-222edf72bed9 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:48.157184 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a4950fb-f4b4-428e-9f27-222edf72bed9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:48.157897 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a4950fb-f4b4-428e-9f27-222edf72bed9 None admin] RBAC: Adding query filter params (group.id=e7758a219cfb46dd9c9560aef5255b8a, role.id=7c1fdac272a34be99661077b0f1b8014, scope.project.id=dd8011750cb34075a6f8106533f848c2) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:48.158203 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a4950fb-f4b4-428e-9f27-222edf72bed9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:48.159638 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a4950fb-f4b4-428e-9f27-222edf72bed9 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:48.165532 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 222/453] 10.222.0.33 () {56 vars in 1376 bytes} [Sun Oct 9 00:42:48 2022] GET /identity/v3/role_assignments?group.id=e7758a219cfb46dd9c9560aef5255b8a&scope.project.id=dd8011750cb34075a6f8106533f848c2&role.id=7c1fdac272a34be99661077b0f1b8014 => generated 611 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:42:48.198852 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-295dca19-6bd2-45ba-a02d-e10cc6970aec None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:48.200537 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:48.236709 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53a9a17c-84ad-42bd-b824-3488bb4596a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:48.237353 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a9a17c-84ad-42bd-b824-3488bb4596a4 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:48.237633 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a9a17c-84ad-42bd-b824-3488bb4596a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:48.237912 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a9a17c-84ad-42bd-b824-3488bb4596a4 None admin] PATH_INFO: `/v3/groups/3026808ce79d453990adc3fe4e692084` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:48.239067 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53a9a17c-84ad-42bd-b824-3488bb4596a4 None admin] RBAC: Authorizing `identity:get_group(group_id=3026808ce79d453990adc3fe4e692084)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:48.240569 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53a9a17c-84ad-42bd-b824-3488bb4596a4 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:48.242181 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 232/454] 10.222.0.33 () {56 vars in 1180 bytes} [Sun Oct 9 00:42:48 2022] GET /identity/v3/groups/3026808ce79d453990adc3fe4e692084 => generated 229 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:42:48.248866 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a4950fb-f4b4-428e-9f27-222edf72bed9 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:48.353375 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a4950fb-f4b4-428e-9f27-222edf72bed9 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:48.387282 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3014dce5-3512-414a-b508-8ae2664ea5c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:48.387921 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3014dce5-3512-414a-b508-8ae2664ea5c0 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:48.388208 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3014dce5-3512-414a-b508-8ae2664ea5c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:48.388483 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3014dce5-3512-414a-b508-8ae2664ea5c0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:48.389202 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3014dce5-3512-414a-b508-8ae2664ea5c0 None admin] RBAC: Adding query filter params (group.id=3026808ce79d453990adc3fe4e692084, role.id=092036ee46ac427baf4d56468575f697, scope.project.id=34e0fbee16724df3abcac7131a4822aa) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:48.389513 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3014dce5-3512-414a-b508-8ae2664ea5c0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:48.390955 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3014dce5-3512-414a-b508-8ae2664ea5c0 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:48.396767 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 223/455] 10.222.0.33 () {56 vars in 1376 bytes} [Sun Oct 9 00:42:48 2022] GET /identity/v3/role_assignments?group.id=3026808ce79d453990adc3fe4e692084&scope.project.id=34e0fbee16724df3abcac7131a4822aa&role.id=092036ee46ac427baf4d56468575f697 => generated 611 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:51.208167 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cef1033-910a-429e-992d-8d8a0f6d088a None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:51.208743 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cef1033-910a-429e-992d-8d8a0f6d088a None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:51.208743 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cef1033-910a-429e-992d-8d8a0f6d088a None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:51.210258 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 233/456] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:42:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:42:51.215743 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ea0c1e-a8fc-4d81-a420-4d3be08d2ebc None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:51.216157 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ea0c1e-a8fc-4d81-a420-4d3be08d2ebc None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:51.216339 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ea0c1e-a8fc-4d81-a420-4d3be08d2ebc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:51.227039 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b353c029-0aae-4365-bd22-d9534ead5e0d None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:51.227323 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b353c029-0aae-4365-bd22-d9534ead5e0d None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:51.227617 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b353c029-0aae-4365-bd22-d9534ead5e0d None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:51.229184 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 234/457] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:42:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 09 00:42:51.234068 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d223d21c-fe87-4e1d-b1ad-f0f4edc9df2a None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:51.234360 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d223d21c-fe87-4e1d-b1ad-f0f4edc9df2a None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:51.234633 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d223d21c-fe87-4e1d-b1ad-f0f4edc9df2a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:51.259006 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-a7ea0c1e-a8fc-4d81-a420-4d3be08d2ebc None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:42:51.260239 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-a7ea0c1e-a8fc-4d81-a420-4d3be08d2ebc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:42:51.266961 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-d223d21c-fe87-4e1d-b1ad-f0f4edc9df2a None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:42:51.268146 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-d223d21c-fe87-4e1d-b1ad-f0f4edc9df2a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:42:51.327817 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:51.383693 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 235/458] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:42:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 09 00:42:51.390997 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 224/459] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:42:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 09 00:42:51.562214 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d20e3f-3290-4628-86b0-21eda724a861 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:51.562596 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d20e3f-3290-4628-86b0-21eda724a861 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:51.562794 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d20e3f-3290-4628-86b0-21eda724a861 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:51.564396 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 236/460] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:42:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:42:51.569589 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e214a4-327f-434d-9ff5-a66d6689ea3b None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:51.569880 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e214a4-327f-434d-9ff5-a66d6689ea3b None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:51.570157 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e214a4-327f-434d-9ff5-a66d6689ea3b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:51.600531 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-79e214a4-327f-434d-9ff5-a66d6689ea3b None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:42:51.601845 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-79e214a4-327f-434d-9ff5-a66d6689ea3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:42:51.685447 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6d20e3f-3290-4628-86b0-21eda724a861 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:51.703787 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 225/461] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:42:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 09 00:42:51.749539 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7aff5f7-49bd-42aa-aa8a-5fde71460a6d None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:51.749862 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7aff5f7-49bd-42aa-aa8a-5fde71460a6d None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:51.750094 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7aff5f7-49bd-42aa-aa8a-5fde71460a6d None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:51.751716 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 226/462] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:42:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 09 00:42:51.754003 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7aff5f7-49bd-42aa-aa8a-5fde71460a6d None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:51.825370 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:51.829825 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6d20e3f-3290-4628-86b0-21eda724a861 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:51.868349 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:51.871776 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bca53c89-fcf6-4a71-976c-22d7401e3047 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:51.872408 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca53c89-fcf6-4a71-976c-22d7401e3047 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:51.872684 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca53c89-fcf6-4a71-976c-22d7401e3047 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:51.872783 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7aff5f7-49bd-42aa-aa8a-5fde71460a6d None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:51.872958 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca53c89-fcf6-4a71-976c-22d7401e3047 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:51.887088 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bca53c89-fcf6-4a71-976c-22d7401e3047 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:51.888590 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bca53c89-fcf6-4a71-976c-22d7401e3047 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:51.903950 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-bca53c89-fcf6-4a71-976c-22d7401e3047 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:42:51.903950 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:42:51.903950 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:42:51.903950 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:42:51.903950 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:42:51.903950 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:42:51.903950 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:42:51.903950 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:42:51.903950 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:42:51.903950 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:42:51.903950 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:42:51.903950 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:42:51.903950 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:42:51.903950 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:42:51.903950 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:42:51.903950 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:42:51.904866 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:42:51.904866 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:42:51.904866 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:42:51.904866 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:42:51.904866 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:42:51.904866 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:42:51.904866 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:42:51.904866 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:42:51.904866 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:42:51.904866 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:42:51.904866 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:42:51.904866 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:42:51.904866 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:42:51.904866 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:42:51.904866 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:42:51.904866 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:42:51.904866 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:42:51.905713 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:42:51.905713 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:42:51.905713 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:42:51.910064 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 237/463] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:42:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 09 00:42:51.911227 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-451b104b-5e52-4af1-a59d-99bacccdf104 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:51.911488 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-451b104b-5e52-4af1-a59d-99bacccdf104 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:51.911788 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-451b104b-5e52-4af1-a59d-99bacccdf104 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:51.917535 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fc20e4b-1591-4fcf-ac82-b17e99fbec43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:51.918423 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc20e4b-1591-4fcf-ac82-b17e99fbec43 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:51.919236 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc20e4b-1591-4fcf-ac82-b17e99fbec43 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:51.919830 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc20e4b-1591-4fcf-ac82-b17e99fbec43 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:51.931016 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-451b104b-5e52-4af1-a59d-99bacccdf104 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:42:51.932120 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-451b104b-5e52-4af1-a59d-99bacccdf104 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:42:51.933400 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fc20e4b-1591-4fcf-ac82-b17e99fbec43 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:51.936198 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fc20e4b-1591-4fcf-ac82-b17e99fbec43 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:51.953656 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-8fc20e4b-1591-4fcf-ac82-b17e99fbec43 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:42:51.953656 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:42:51.953656 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:42:51.953656 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:42:51.953656 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:42:51.953656 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:42:51.953656 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:42:51.953656 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:42:51.953656 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:42:51.953656 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:42:51.953656 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:42:51.953656 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:42:51.953656 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:42:51.953656 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:42:51.953656 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:42:51.953656 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:42:51.954544 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:42:51.954544 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:42:51.954544 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:42:51.954544 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:42:51.954544 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:42:51.954544 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:42:51.954544 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:42:51.954544 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:42:51.954544 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:42:51.954544 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:42:51.954544 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:42:51.954544 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:42:51.954544 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:42:51.954544 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:42:51.954544 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:42:51.954544 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:42:51.954544 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:42:51.955392 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:42:51.955392 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:42:51.955392 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:42:51.956993 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 227/464] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:42:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:42:51.962038 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fc20e4b-1591-4fcf-ac82-b17e99fbec43 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:52.022215 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 238/465] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:42:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 09 00:42:52.023951 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-451b104b-5e52-4af1-a59d-99bacccdf104 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:52.100147 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fc20e4b-1591-4fcf-ac82-b17e99fbec43 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:52.133784 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:52.149279 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac1acdfc-2415-4d4b-bbae-96d686445653 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:52.149989 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac1acdfc-2415-4d4b-bbae-96d686445653 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:52.150578 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac1acdfc-2415-4d4b-bbae-96d686445653 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:52.150925 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac1acdfc-2415-4d4b-bbae-96d686445653 None admin] PATH_INFO: `/v3/domains` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:52.152059 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac1acdfc-2415-4d4b-bbae-96d686445653 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:52.152439 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac1acdfc-2415-4d4b-bbae-96d686445653 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:52.154165 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac1acdfc-2415-4d4b-bbae-96d686445653 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:52.154997 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-451b104b-5e52-4af1-a59d-99bacccdf104 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:52.171354 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 228/466] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:42:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:52.172600 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac1acdfc-2415-4d4b-bbae-96d686445653 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:52.192368 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:52.193093 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4ff342f-df5e-40b4-a557-6c06260616a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:52.193966 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ff342f-df5e-40b4-a557-6c06260616a6 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:52.194483 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ff342f-df5e-40b4-a557-6c06260616a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:52.194962 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ff342f-df5e-40b4-a557-6c06260616a6 None admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:52.195953 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4ff342f-df5e-40b4-a557-6c06260616a6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:52.196473 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4ff342f-df5e-40b4-a557-6c06260616a6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:52.198037 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4ff342f-df5e-40b4-a557-6c06260616a6 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:52.215915 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 239/467] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:42:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:52.217486 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4ff342f-df5e-40b4-a557-6c06260616a6 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:52.296071 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac1acdfc-2415-4d4b-bbae-96d686445653 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:52.339069 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe212cd7-e242-498d-a1c0-430af5b95a1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:52.339754 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe212cd7-e242-498d-a1c0-430af5b95a1d None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:52.340035 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe212cd7-e242-498d-a1c0-430af5b95a1d None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:52.340305 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe212cd7-e242-498d-a1c0-430af5b95a1d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:52.341667 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4ff342f-df5e-40b4-a557-6c06260616a6 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:52.351401 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe212cd7-e242-498d-a1c0-430af5b95a1d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:52.353548 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe212cd7-e242-498d-a1c0-430af5b95a1d None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:52.369107 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-fe212cd7-e242-498d-a1c0-430af5b95a1d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:42:52.369107 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:42:52.369107 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:42:52.369107 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:42:52.369107 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:42:52.369107 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:42:52.369107 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:42:52.369107 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:42:52.369107 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:42:52.369107 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:42:52.369107 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:42:52.369107 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:42:52.369107 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:42:52.369107 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:42:52.369107 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:42:52.369107 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:42:52.369986 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:42:52.369986 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:42:52.369986 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:42:52.369986 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:42:52.369986 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:42:52.369986 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:42:52.369986 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:42:52.369986 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:42:52.369986 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:42:52.369986 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:42:52.369986 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:42:52.369986 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:42:52.369986 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:42:52.369986 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:42:52.369986 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:42:52.369986 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:42:52.369986 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:42:52.370826 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:42:52.370826 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:42:52.370826 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:42:52.370826 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 229/468] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:42:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:42:52.371051 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe212cd7-e242-498d-a1c0-430af5b95a1d None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:52.387674 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0354a985-9e66-487f-96ce-780a953bf362 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:52.388283 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0354a985-9e66-487f-96ce-780a953bf362 None admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:52.388593 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0354a985-9e66-487f-96ce-780a953bf362 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:52.388867 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0354a985-9e66-487f-96ce-780a953bf362 None admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:52.389512 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0354a985-9e66-487f-96ce-780a953bf362 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:52.390939 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0354a985-9e66-487f-96ce-780a953bf362 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:52.407429 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:52.412683 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 240/469] 10.222.0.33 () {60 vars in 1132 bytes} [Sun Oct 9 00:42:52 2022] POST /identity/v3/users => generated 246 bytes in 195 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:42:52.413776 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0354a985-9e66-487f-96ce-780a953bf362 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:52.465243 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:52.510818 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe212cd7-e242-498d-a1c0-430af5b95a1d None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:52.547826 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f28d9da-daa2-43d8-aee9-f8e627fb6279 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:52.548448 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f28d9da-daa2-43d8-aee9-f8e627fb6279 None admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:52.548728 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f28d9da-daa2-43d8-aee9-f8e627fb6279 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:52.548997 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f28d9da-daa2-43d8-aee9-f8e627fb6279 None admin] PATH_INFO: `/v3/users` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:52.549646 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f28d9da-daa2-43d8-aee9-f8e627fb6279 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:52.551107 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f28d9da-daa2-43d8-aee9-f8e627fb6279 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:52.553799 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0354a985-9e66-487f-96ce-780a953bf362 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:52.574604 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 230/470] 10.222.0.33 () {60 vars in 1132 bytes} [Sun Oct 9 00:42:52 2022] POST /identity/v3/users => generated 248 bytes in 204 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:42:52.575520 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f28d9da-daa2-43d8-aee9-f8e627fb6279 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:52.605013 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b4830cf-4c00-4507-9bc9-7af559020df4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:52.605844 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b4830cf-4c00-4507-9bc9-7af559020df4 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:52.606123 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b4830cf-4c00-4507-9bc9-7af559020df4 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:52.606394 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b4830cf-4c00-4507-9bc9-7af559020df4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:52.624692 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b4830cf-4c00-4507-9bc9-7af559020df4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:52.626198 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b4830cf-4c00-4507-9bc9-7af559020df4 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:52.645541 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-1b4830cf-4c00-4507-9bc9-7af559020df4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:42:52.645541 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:42:52.645541 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:42:52.645541 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:42:52.645541 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:42:52.645541 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:42:52.645541 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:42:52.645541 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:42:52.645541 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:42:52.645541 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:42:52.645541 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:42:52.645541 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:42:52.645541 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:42:52.645541 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:42:52.645541 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:42:52.645541 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:42:52.646691 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:42:52.646691 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:42:52.646691 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:42:52.646691 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:42:52.646691 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:42:52.646691 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:42:52.646691 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:42:52.646691 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:42:52.646691 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:42:52.646691 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:42:52.646691 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:42:52.646691 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:42:52.646691 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:42:52.646691 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:42:52.646691 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:42:52.646691 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:42:52.646691 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:42:52.647546 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:42:52.647546 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:42:52.647546 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:42:52.649565 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 241/471] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:42:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 236 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:42:52.650814 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ba0b8a0-1b28-4220-80dd-309c4694dbb2 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:52.651092 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ba0b8a0-1b28-4220-80dd-309c4694dbb2 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:52.651362 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ba0b8a0-1b28-4220-80dd-309c4694dbb2 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:52.652896 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 242/472] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:42:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:42:52.657377 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ba0b8a0-1b28-4220-80dd-309c4694dbb2 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:52.724947 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:52.729259 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f28d9da-daa2-43d8-aee9-f8e627fb6279 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:52.756381 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:52.780894 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00dcd953-f569-49b0-b2f0-5cbcac440488 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:52.781510 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00dcd953-f569-49b0-b2f0-5cbcac440488 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:52.781790 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00dcd953-f569-49b0-b2f0-5cbcac440488 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:52.782064 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00dcd953-f569-49b0-b2f0-5cbcac440488 None admin] PATH_INFO: `/v3/domains` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:52.782708 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00dcd953-f569-49b0-b2f0-5cbcac440488 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:52.783128 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00dcd953-f569-49b0-b2f0-5cbcac440488 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:52.785220 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00dcd953-f569-49b0-b2f0-5cbcac440488 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:52.803686 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 231/473] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:42:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:52.804614 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ba0b8a0-1b28-4220-80dd-309c4694dbb2 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:52.807405 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08aa866c-124e-473b-aafa-22da3e3137b3 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:52.807724 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08aa866c-124e-473b-aafa-22da3e3137b3 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:52.807998 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08aa866c-124e-473b-aafa-22da3e3137b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:52.826813 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-08aa866c-124e-473b-aafa-22da3e3137b3 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:42:52.828571 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-08aa866c-124e-473b-aafa-22da3e3137b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:42:52.847265 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32bb1ade-98a0-4d0b-a7e3-2e935fd20d57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:52.847911 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32bb1ade-98a0-4d0b-a7e3-2e935fd20d57 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:52.848189 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32bb1ade-98a0-4d0b-a7e3-2e935fd20d57 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:52.848477 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32bb1ade-98a0-4d0b-a7e3-2e935fd20d57 None admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:52.849100 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32bb1ade-98a0-4d0b-a7e3-2e935fd20d57 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:52.849398 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32bb1ade-98a0-4d0b-a7e3-2e935fd20d57 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:52.851013 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32bb1ade-98a0-4d0b-a7e3-2e935fd20d57 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:52.865533 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 243/474] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:42:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:42:52.866758 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32bb1ade-98a0-4d0b-a7e3-2e935fd20d57 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:52.913312 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 232/475] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:42:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 09 00:42:52.915067 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08aa866c-124e-473b-aafa-22da3e3137b3 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:52.995019 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32bb1ade-98a0-4d0b-a7e3-2e935fd20d57 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:53.015474 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:53.046322 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08aa866c-124e-473b-aafa-22da3e3137b3 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:53.059716 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:53.082031 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6057316c-15a8-4712-8123-417bb4ac5aaa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:53.082811 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6057316c-15a8-4712-8123-417bb4ac5aaa None admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:53.083199 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6057316c-15a8-4712-8123-417bb4ac5aaa None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:53.083763 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6057316c-15a8-4712-8123-417bb4ac5aaa None admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:53.084397 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6057316c-15a8-4712-8123-417bb4ac5aaa None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:53.086552 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6057316c-15a8-4712-8123-417bb4ac5aaa None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:53.089377 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed950df9-82ba-458f-869e-a0d7b5210744 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:53.090230 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed950df9-82ba-458f-869e-a0d7b5210744 None admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:53.090699 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed950df9-82ba-458f-869e-a0d7b5210744 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:53.091159 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed950df9-82ba-458f-869e-a0d7b5210744 None admin] PATH_INFO: `/v3/users` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:53.092231 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed950df9-82ba-458f-869e-a0d7b5210744 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:53.093843 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed950df9-82ba-458f-869e-a0d7b5210744 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:53.126164 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 233/476] 10.222.0.33 () {60 vars in 1132 bytes} [Sun Oct 9 00:42:52 2022] POST /identity/v3/users => generated 249 bytes in 212 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:42:53.142586 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 244/477] 10.222.0.33 () {60 vars in 1132 bytes} [Sun Oct 9 00:42:52 2022] POST /identity/v3/users => generated 248 bytes in 270 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:42:53.269468 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed950df9-82ba-458f-869e-a0d7b5210744 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:53.404233 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed950df9-82ba-458f-869e-a0d7b5210744 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:53.443476 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-859ebdf3-a447-4754-b732-cb5bc582f900 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:53.444162 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-859ebdf3-a447-4754-b732-cb5bc582f900 None admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:53.444446 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-859ebdf3-a447-4754-b732-cb5bc582f900 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:53.444724 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-859ebdf3-a447-4754-b732-cb5bc582f900 None admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:53.445450 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-859ebdf3-a447-4754-b732-cb5bc582f900 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:53.446842 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-859ebdf3-a447-4754-b732-cb5bc582f900 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:53.456592 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.sql.core [None req-859ebdf3-a447-4754-b732-cb5bc582f900 None admin] Conflict role: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Oct 09 00:42:53.456592 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Oct 09 00:42:53.456592 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [parameters: {'id': '1d932c52cd6d4539ae2a708f9582b6e1', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Oct 09 00:42:53.456592 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81453) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 09 00:42:53.473478 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-859ebdf3-a447-4754-b732-cb5bc582f900 None admin] Conflict occurred attempting to store role - Duplicate entry found with name anotherrole.: keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Oct 09 00:42:53.473478 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:42:53.473478 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 09 00:42:53.473478 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 09 00:42:53.473478 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 09 00:42:53.473478 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 09 00:42:53.473478 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 09 00:42:53.473478 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application result = self._query(query) Oct 09 00:42:53.473478 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 09 00:42:53.473478 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application conn.query(q) Oct 09 00:42:53.473478 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 09 00:42:53.473478 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 09 00:42:53.473478 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 09 00:42:53.473478 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application result.read() Oct 09 00:42:53.473478 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 09 00:42:53.474360 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 09 00:42:53.474360 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 09 00:42:53.474360 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 09 00:42:53.474360 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 09 00:42:53.474360 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 09 00:42:53.474360 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 09 00:42:53.474360 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 09 00:42:53.474360 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Oct 09 00:42:53.474360 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:42:53.474360 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 09 00:42:53.474360 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:42:53.474360 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:42:53.474360 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 09 00:42:53.474360 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 09 00:42:53.474360 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 33, in create_role Oct 09 00:42:53.474360 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return ref.to_dict() Oct 09 00:42:53.474360 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 09 00:42:53.474360 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application next(self.gen) Oct 09 00:42:53.475741 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Oct 09 00:42:53.475741 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application yield resource Oct 09 00:42:53.475741 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 09 00:42:53.475741 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application next(self.gen) Oct 09 00:42:53.475741 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Oct 09 00:42:53.475741 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application self.session.rollback() Oct 09 00:42:53.475741 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 09 00:42:53.475741 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application self.force_reraise() Oct 09 00:42:53.475741 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 09 00:42:53.475741 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise self.value Oct 09 00:42:53.475741 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Oct 09 00:42:53.475741 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 09 00:42:53.475741 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Oct 09 00:42:53.475741 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application session.commit() Oct 09 00:42:53.475741 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 09 00:42:53.475741 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 09 00:42:53.475741 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 09 00:42:53.477505 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application self._prepare_impl() Oct 09 00:42:53.477505 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 09 00:42:53.477505 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application self.session.flush() Oct 09 00:42:53.477505 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 09 00:42:53.477505 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application self._flush(objects) Oct 09 00:42:53.477505 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 09 00:42:53.477505 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 09 00:42:53.477505 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 09 00:42:53.477505 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application compat.raise_( Oct 09 00:42:53.477505 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 09 00:42:53.477505 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception Oct 09 00:42:53.477505 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 09 00:42:53.477505 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application flush_context.execute() Oct 09 00:42:53.477505 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 09 00:42:53.477505 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rec.execute(self) Oct 09 00:42:53.477505 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 09 00:42:53.477505 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 09 00:42:53.478608 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 09 00:42:53.478608 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 09 00:42:53.478608 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Oct 09 00:42:53.478608 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application c = connection._execute_20( Oct 09 00:42:53.478608 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 09 00:42:53.478608 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 09 00:42:53.478608 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 09 00:42:53.478608 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 09 00:42:53.478608 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 09 00:42:53.478608 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 09 00:42:53.478608 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 09 00:42:53.478608 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 09 00:42:53.478608 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 09 00:42:53.478608 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 09 00:42:53.478608 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 09 00:42:53.478608 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception Oct 09 00:42:53.479401 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 09 00:42:53.479401 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 09 00:42:53.479401 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 09 00:42:53.479401 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 09 00:42:53.479401 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 09 00:42:53.479401 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application result = self._query(query) Oct 09 00:42:53.479401 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 09 00:42:53.479401 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application conn.query(q) Oct 09 00:42:53.479401 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 09 00:42:53.479401 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 09 00:42:53.479401 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 09 00:42:53.479401 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application result.read() Oct 09 00:42:53.479401 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 09 00:42:53.479401 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 09 00:42:53.479401 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 09 00:42:53.479401 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 09 00:42:53.480296 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 09 00:42:53.480296 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 09 00:42:53.480296 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 09 00:42:53.480296 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 09 00:42:53.480296 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Oct 09 00:42:53.480296 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Oct 09 00:42:53.480296 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [parameters: {'id': '1d932c52cd6d4539ae2a708f9582b6e1', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Oct 09 00:42:53.480296 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 09 00:42:53.480296 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:42:53.480296 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 09 00:42:53.480296 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:42:53.480296 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:42:53.480296 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:42:53.480296 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:42:53.480296 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:42:53.480296 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:42:53.485398 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:42:53.485398 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:42:53.485398 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:42:53.485398 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:42:53.485398 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:42:53.485398 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:42:53.485398 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 115, in post Oct 09 00:42:53.485398 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ref = PROVIDERS.role_api.create_role( Oct 09 00:42:53.485398 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:42:53.485398 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:42:53.485398 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1298, in create_role Oct 09 00:42:53.485398 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ret = self.driver.create_role(role_id, role) Oct 09 00:42:53.485398 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 09 00:42:53.485398 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 09 00:42:53.485398 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Oct 09 00:42:53.485398 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:42:53.486212 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 234/478] 10.222.0.33 () {60 vars in 1131 bytes} [Sun Oct 9 00:42:53 2022] POST /identity/v3/roles => generated 144 bytes in 206 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:42:53.486277 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6057316c-15a8-4712-8123-417bb4ac5aaa None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:53.597073 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6057316c-15a8-4712-8123-417bb4ac5aaa None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:53.644348 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d5a1571-9ed5-4a81-a76e-99f0cea50381 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:53.644995 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d5a1571-9ed5-4a81-a76e-99f0cea50381 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:53.645275 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d5a1571-9ed5-4a81-a76e-99f0cea50381 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:53.645548 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d5a1571-9ed5-4a81-a76e-99f0cea50381 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:53.657625 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d5a1571-9ed5-4a81-a76e-99f0cea50381 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:42:53.662818 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-3d5a1571-9ed5-4a81-a76e-99f0cea50381 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 09 00:42:53.662818 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:42:53.662818 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:42:53.662818 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:42:53.662818 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:42:53.662818 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:42:53.662818 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:42:53.662818 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:42:53.662818 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:42:53.662818 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:42:53.662818 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:42:53.662818 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:42:53.662818 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:42:53.662818 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:42:53.662818 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:42:53.662818 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:42:53.664584 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:42:53.664584 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:42:53.664584 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:42:53.664584 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:42:53.664584 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:42:53.664584 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:42:53.664584 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:42:53.664584 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:42:53.664584 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:42:53.664584 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:42:53.664584 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:42:53.664584 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:42:53.664584 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:42:53.664584 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:42:53.664584 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:42:53.664584 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:42:53.665771 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:42:53.665771 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:42:53.665771 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:42:53.665771 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:42:53.665771 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:42:53.665771 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:42:53.665771 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:42:53.665771 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:42:53.665771 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:42:53.665771 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:42:53.665771 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 09 00:42:53.665771 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:42:53.665771 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 245/479] 10.222.0.33 () {56 vars in 1114 bytes} [Sun Oct 9 00:42:53 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:42:53.671037 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-859ebdf3-a447-4754-b732-cb5bc582f900 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:53.705098 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:53.778826 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-859ebdf3-a447-4754-b732-cb5bc582f900 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:53.816989 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0118b594-5b2c-4387-a776-728b432dd5dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:53.817597 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0118b594-5b2c-4387-a776-728b432dd5dc None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:53.817876 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0118b594-5b2c-4387-a776-728b432dd5dc None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:53.818148 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0118b594-5b2c-4387-a776-728b432dd5dc None admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:53.818817 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0118b594-5b2c-4387-a776-728b432dd5dc None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:53.819122 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0118b594-5b2c-4387-a776-728b432dd5dc None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:53.821347 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0118b594-5b2c-4387-a776-728b432dd5dc None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:53.830324 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 235/480] 10.222.0.33 () {56 vars in 1111 bytes} [Sun Oct 9 00:42:53 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:54.858385 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da77e9a2-0a46-4108-9779-022027d3f5e9 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:54.858903 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da77e9a2-0a46-4108-9779-022027d3f5e9 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:54.858903 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da77e9a2-0a46-4108-9779-022027d3f5e9 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:54.860504 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 246/481] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:42:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:42:54.865953 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8751c1e-aaca-4a7b-939e-99671d9f9576 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:54.866332 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8751c1e-aaca-4a7b-939e-99671d9f9576 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:54.866507 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8751c1e-aaca-4a7b-939e-99671d9f9576 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:54.892404 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-d8751c1e-aaca-4a7b-939e-99671d9f9576 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:42:54.893663 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-d8751c1e-aaca-4a7b-939e-99671d9f9576 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:42:54.990184 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 236/482] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:42:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 09 00:42:55.157662 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c2db0b-e843-4d2e-9178-02a38f14b515 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:55.157950 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c2db0b-e843-4d2e-9178-02a38f14b515 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:55.158126 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c2db0b-e843-4d2e-9178-02a38f14b515 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:55.159816 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 247/483] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:42:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:42:55.165846 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc87cfc4-3794-473b-a5ea-1edc8db6c01b None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:55.166139 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc87cfc4-3794-473b-a5ea-1edc8db6c01b None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:55.166415 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc87cfc4-3794-473b-a5ea-1edc8db6c01b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:55.191692 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-bc87cfc4-3794-473b-a5ea-1edc8db6c01b None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:42:55.192886 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-bc87cfc4-3794-473b-a5ea-1edc8db6c01b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:42:55.210612 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:55.276400 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 237/484] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:42:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 09 00:42:55.391965 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04c2db0b-e843-4d2e-9178-02a38f14b515 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:55.420320 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:55.517305 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04c2db0b-e843-4d2e-9178-02a38f14b515 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:55.560443 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad3e43a-6bb9-4b08-99be-24b731a7768d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:55.561112 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad3e43a-6bb9-4b08-99be-24b731a7768d None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:55.561405 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad3e43a-6bb9-4b08-99be-24b731a7768d None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:55.561679 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad3e43a-6bb9-4b08-99be-24b731a7768d None admin] PATH_INFO: `/v3/roles/service` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:55.580521 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dad3e43a-6bb9-4b08-99be-24b731a7768d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:42:55.585292 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-dad3e43a-6bb9-4b08-99be-24b731a7768d None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 09 00:42:55.585292 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:42:55.585292 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:42:55.585292 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:42:55.585292 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:42:55.585292 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:42:55.585292 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:42:55.585292 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:42:55.585292 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:42:55.585292 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:42:55.585292 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:42:55.585292 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:42:55.585292 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:42:55.585292 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:42:55.585292 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:42:55.585292 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:42:55.586206 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:42:55.586206 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:42:55.586206 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:42:55.586206 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:42:55.586206 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:42:55.586206 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:42:55.586206 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:42:55.586206 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:42:55.586206 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:42:55.586206 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:42:55.586206 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:42:55.586206 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:42:55.586206 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:42:55.586206 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:42:55.586206 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:42:55.586206 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:42:55.587556 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:42:55.587556 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:42:55.587556 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:42:55.587556 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:42:55.587556 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:42:55.587556 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:42:55.587556 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:42:55.587556 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:42:55.587556 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:42:55.587556 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:42:55.587556 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 09 00:42:55.587556 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:42:55.587556 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 248/485] 10.222.0.33 () {56 vars in 1102 bytes} [Sun Oct 9 00:42:55 2022] GET /identity/v3/roles/service => generated 85 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:42:55.592212 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc87cfc4-3794-473b-a5ea-1edc8db6c01b None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:55.664014 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad3e43a-6bb9-4b08-99be-24b731a7768d None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:55.748535 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc87cfc4-3794-473b-a5ea-1edc8db6c01b None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:55.806089 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:55.810075 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dd8ce68-36c7-4c8d-b35a-5eadc0315f5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:55.810938 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dd8ce68-36c7-4c8d-b35a-5eadc0315f5e None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:55.810938 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dd8ce68-36c7-4c8d-b35a-5eadc0315f5e None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:55.811210 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dd8ce68-36c7-4c8d-b35a-5eadc0315f5e None admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:55.811898 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dd8ce68-36c7-4c8d-b35a-5eadc0315f5e None admin] RBAC: Adding query filter params (name=service) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:55.812277 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dd8ce68-36c7-4c8d-b35a-5eadc0315f5e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:55.813729 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dd8ce68-36c7-4c8d-b35a-5eadc0315f5e None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:55.827605 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 238/486] 10.222.0.33 () {56 vars in 1103 bytes} [Sun Oct 9 00:42:55 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:55.833092 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dd8ce68-36c7-4c8d-b35a-5eadc0315f5e None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:55.854300 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad3e43a-6bb9-4b08-99be-24b731a7768d None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:55.876993 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:55.940437 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82cdfbaf-4804-486e-a6ee-ce34a29912f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:55.941089 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82cdfbaf-4804-486e-a6ee-ce34a29912f2 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:55.941333 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82cdfbaf-4804-486e-a6ee-ce34a29912f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:55.941603 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82cdfbaf-4804-486e-a6ee-ce34a29912f2 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:55.967461 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82cdfbaf-4804-486e-a6ee-ce34a29912f2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:42:55.972753 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-82cdfbaf-4804-486e-a6ee-ce34a29912f2 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 09 00:42:55.972753 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:42:55.972753 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:42:55.972753 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:42:55.972753 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:42:55.972753 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:42:55.972753 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:42:55.972753 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:42:55.972753 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:42:55.972753 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:42:55.972753 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:42:55.972753 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:42:55.972753 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:42:55.972753 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:42:55.972753 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:42:55.972753 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:42:55.973649 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:42:55.973649 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:42:55.973649 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:42:55.973649 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:42:55.973649 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:42:55.973649 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:42:55.973649 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:42:55.973649 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:42:55.973649 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:42:55.973649 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:42:55.973649 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:42:55.973649 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:42:55.973649 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:42:55.973649 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:42:55.973649 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:42:55.973649 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:42:55.974433 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:42:55.974433 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:42:55.974433 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:42:55.974433 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:42:55.974433 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:42:55.974433 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:42:55.974433 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:42:55.974433 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:42:55.974433 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:42:55.974433 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:42:55.974433 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 09 00:42:55.974433 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:42:55.974433 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 249/487] 10.222.0.33 () {56 vars in 1102 bytes} [Sun Oct 9 00:42:55 2022] GET /identity/v3/roles/service => generated 85 bytes in 311 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:42:55.980546 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82cdfbaf-4804-486e-a6ee-ce34a29912f2 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:56.022642 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dd8ce68-36c7-4c8d-b35a-5eadc0315f5e None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:56.100348 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb6b8e21-c24c-4b6a-b148-4f1f086148fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:56.101002 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6b8e21-c24c-4b6a-b148-4f1f086148fe None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:56.101286 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6b8e21-c24c-4b6a-b148-4f1f086148fe None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:56.102148 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6b8e21-c24c-4b6a-b148-4f1f086148fe None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:56.125357 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb6b8e21-c24c-4b6a-b148-4f1f086148fe None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:56.127182 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb6b8e21-c24c-4b6a-b148-4f1f086148fe None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:56.145555 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-cb6b8e21-c24c-4b6a-b148-4f1f086148fe None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:42:56.145555 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:42:56.145555 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:42:56.145555 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:42:56.145555 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:42:56.145555 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:42:56.145555 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:42:56.145555 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:42:56.145555 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:42:56.145555 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:42:56.145555 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:42:56.145555 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:42:56.145555 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:42:56.145555 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:42:56.145555 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:42:56.145555 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:42:56.146487 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:42:56.146487 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:42:56.146487 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:42:56.146487 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:42:56.146487 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:42:56.146487 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:42:56.146487 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:42:56.146487 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:42:56.146487 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:42:56.146487 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:42:56.146487 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:42:56.146487 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:42:56.146487 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:42:56.146487 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:42:56.146487 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:42:56.146487 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:42:56.146487 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:42:56.147303 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:42:56.147303 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:42:56.147303 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:42:56.149666 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 239/488] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:42:55 2022] GET /identity/v3/domains/Default => generated 87 bytes in 317 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:42:56.151100 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a11d3a-7406-487e-96d5-24edfe2bbc04 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:56.151586 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a11d3a-7406-487e-96d5-24edfe2bbc04 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:56.152182 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a11d3a-7406-487e-96d5-24edfe2bbc04 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:56.153749 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 240/489] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:42:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:42:56.156087 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82cdfbaf-4804-486e-a6ee-ce34a29912f2 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:56.159355 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a11d3a-7406-487e-96d5-24edfe2bbc04 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:56.192984 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c63dbac5-2970-4195-9fd9-8c4badcef2ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:56.193600 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c63dbac5-2970-4195-9fd9-8c4badcef2ca None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:56.193883 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c63dbac5-2970-4195-9fd9-8c4badcef2ca None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:56.194164 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c63dbac5-2970-4195-9fd9-8c4badcef2ca None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:56.195099 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c63dbac5-2970-4195-9fd9-8c4badcef2ca None admin] RBAC: Adding query filter params (name=service) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:56.195499 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c63dbac5-2970-4195-9fd9-8c4badcef2ca None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:56.197050 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c63dbac5-2970-4195-9fd9-8c4badcef2ca None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:56.206440 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 250/490] 10.222.0.33 () {56 vars in 1103 bytes} [Sun Oct 9 00:42:55 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:56.207825 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c4b6c2-1253-4ea3-89c4-abd49e1014af None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:56.208110 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c4b6c2-1253-4ea3-89c4-abd49e1014af None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:56.208415 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c4b6c2-1253-4ea3-89c4-abd49e1014af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:56.228361 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-28c4b6c2-1253-4ea3-89c4-abd49e1014af None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:42:56.229330 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:56.230706 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-28c4b6c2-1253-4ea3-89c4-abd49e1014af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:42:56.232889 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:56.269895 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a11d3a-7406-487e-96d5-24edfe2bbc04 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:56.305927 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 251/491] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:42:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 09 00:42:56.306492 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28c4b6c2-1253-4ea3-89c4-abd49e1014af None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:56.312066 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3e3d330-88cb-450b-a61a-b227241b61a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:56.312702 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e3d330-88cb-450b-a61a-b227241b61a3 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:56.312991 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e3d330-88cb-450b-a61a-b227241b61a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:56.313268 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e3d330-88cb-450b-a61a-b227241b61a3 None admin] PATH_INFO: `/v3/domains` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:56.313912 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3e3d330-88cb-450b-a61a-b227241b61a3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:56.314216 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3e3d330-88cb-450b-a61a-b227241b61a3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:56.315655 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3e3d330-88cb-450b-a61a-b227241b61a3 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:56.331973 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 241/492] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:42:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:56.333633 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca831e7-e444-473a-9097-928884dee569 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:56.334133 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca831e7-e444-473a-9097-928884dee569 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:56.334593 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca831e7-e444-473a-9097-928884dee569 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:56.336957 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 242/493] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:42:56 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:42:56.338004 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ca831e7-e444-473a-9097-928884dee569 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:56.425068 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28c4b6c2-1253-4ea3-89c4-abd49e1014af None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:56.462756 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ca831e7-e444-473a-9097-928884dee569 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:56.463174 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28bce158-e154-42f3-92b3-ae58c041d4c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:56.463888 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28bce158-e154-42f3-92b3-ae58c041d4c6 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:56.464175 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28bce158-e154-42f3-92b3-ae58c041d4c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:56.464450 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28bce158-e154-42f3-92b3-ae58c041d4c6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:56.473242 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:56.478680 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28bce158-e154-42f3-92b3-ae58c041d4c6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:56.481238 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28bce158-e154-42f3-92b3-ae58c041d4c6 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:56.496469 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:56.502948 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-28bce158-e154-42f3-92b3-ae58c041d4c6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:42:56.502948 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:42:56.502948 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:42:56.502948 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:42:56.502948 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:42:56.502948 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:42:56.502948 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:42:56.502948 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:42:56.502948 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:42:56.502948 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:42:56.502948 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:42:56.502948 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:42:56.502948 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:42:56.502948 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:42:56.502948 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:42:56.502948 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:42:56.503847 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:42:56.503847 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:42:56.503847 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:42:56.503847 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:42:56.503847 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:42:56.503847 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:42:56.503847 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:42:56.503847 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:42:56.503847 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:42:56.503847 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:42:56.503847 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:42:56.503847 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:42:56.503847 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:42:56.503847 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:42:56.503847 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:42:56.503847 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:42:56.503847 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:42:56.504898 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:42:56.504898 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:42:56.504898 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:42:56.504898 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 252/494] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:42:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:42:56.506249 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a424d818-2bba-4c31-a0df-30bf3a1f6b0b None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:56.506537 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a424d818-2bba-4c31-a0df-30bf3a1f6b0b None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:56.506812 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a424d818-2bba-4c31-a0df-30bf3a1f6b0b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:56.510272 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cb1d58b-74dd-4926-88f3-b70ff45b8f29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:56.511238 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cb1d58b-74dd-4926-88f3-b70ff45b8f29 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:56.511765 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cb1d58b-74dd-4926-88f3-b70ff45b8f29 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:56.512270 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cb1d58b-74dd-4926-88f3-b70ff45b8f29 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:56.526010 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cb1d58b-74dd-4926-88f3-b70ff45b8f29 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:56.527815 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cb1d58b-74dd-4926-88f3-b70ff45b8f29 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:56.527987 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-a424d818-2bba-4c31-a0df-30bf3a1f6b0b None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:42:56.529495 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-a424d818-2bba-4c31-a0df-30bf3a1f6b0b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:42:56.543518 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-3cb1d58b-74dd-4926-88f3-b70ff45b8f29 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 09 00:42:56.543518 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:42:56.543518 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:42:56.543518 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:42:56.543518 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:42:56.543518 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:42:56.543518 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:42:56.543518 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:42:56.543518 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:42:56.543518 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:42:56.543518 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:42:56.543518 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:42:56.543518 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 09 00:42:56.543518 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 09 00:42:56.543518 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 09 00:42:56.543518 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 09 00:42:56.544416 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:42:56.544416 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:42:56.544416 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 09 00:42:56.544416 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 09 00:42:56.544416 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 09 00:42:56.544416 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 09 00:42:56.544416 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:42:56.544416 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:42:56.544416 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:42:56.544416 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:42:56.544416 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:42:56.544416 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:42:56.544416 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:42:56.544416 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:42:56.544416 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:42:56.544416 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:42:56.544416 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:42:56.544416 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:42:56.545268 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:42:56.545268 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:42:56.545268 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 09 00:42:56.545268 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 09 00:42:56.545268 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 09 00:42:56.545268 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 09 00:42:56.545268 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 09 00:42:56.545268 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 09 00:42:56.545268 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 09 00:42:56.545268 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:42:56.548071 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 243/495] 10.222.0.33 () {56 vars in 1093 bytes} [Sun Oct 9 00:42:56 2022] GET /identity/v3/users/nova => generated 82 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:42:56.549100 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cb1d58b-74dd-4926-88f3-b70ff45b8f29 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:56.626680 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 253/496] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:42:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 09 00:42:56.627447 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a424d818-2bba-4c31-a0df-30bf3a1f6b0b None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:56.676354 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cb1d58b-74dd-4926-88f3-b70ff45b8f29 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:56.717080 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bdec608-7d7e-4587-9dee-64f8f588eece None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:56.717481 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bdec608-7d7e-4587-9dee-64f8f588eece None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:56.717763 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bdec608-7d7e-4587-9dee-64f8f588eece None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:56.718037 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bdec608-7d7e-4587-9dee-64f8f588eece None admin] PATH_INFO: `/v3/domains` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:56.719007 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bdec608-7d7e-4587-9dee-64f8f588eece None admin] RBAC: Adding query filter params (name=Default) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:56.719111 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bdec608-7d7e-4587-9dee-64f8f588eece None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:56.720674 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bdec608-7d7e-4587-9dee-64f8f588eece None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:56.738125 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 244/497] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:42:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:56.738125 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bdec608-7d7e-4587-9dee-64f8f588eece None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:56.742277 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a424d818-2bba-4c31-a0df-30bf3a1f6b0b None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:56.763877 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:56.779582 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aed80006-aa06-4712-89c1-3664e6901b63 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:56.780458 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed80006-aa06-4712-89c1-3664e6901b63 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:56.780745 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed80006-aa06-4712-89c1-3664e6901b63 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:56.781015 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed80006-aa06-4712-89c1-3664e6901b63 None admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:56.781734 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aed80006-aa06-4712-89c1-3664e6901b63 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:56.782038 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aed80006-aa06-4712-89c1-3664e6901b63 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:56.783475 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aed80006-aa06-4712-89c1-3664e6901b63 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:56.793285 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:56.824234 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 254/498] 10.222.0.33 () {56 vars in 1133 bytes} [Sun Oct 9 00:42:56 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:56.824234 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7bf50e2-8512-4c69-81b4-1c09ad0e72a4 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:56.824234 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7bf50e2-8512-4c69-81b4-1c09ad0e72a4 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:56.824234 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7bf50e2-8512-4c69-81b4-1c09ad0e72a4 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:56.824234 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 255/499] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:42:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:42:56.824638 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7bf50e2-8512-4c69-81b4-1c09ad0e72a4 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:56.854898 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bdec608-7d7e-4587-9dee-64f8f588eece None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:56.890411 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a99f04e-b98a-41b5-9f4d-5a8c01293876 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:56.890998 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a99f04e-b98a-41b5-9f4d-5a8c01293876 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:56.891226 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a99f04e-b98a-41b5-9f4d-5a8c01293876 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:56.891507 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a99f04e-b98a-41b5-9f4d-5a8c01293876 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:56.903676 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a99f04e-b98a-41b5-9f4d-5a8c01293876 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:42:56.908509 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-2a99f04e-b98a-41b5-9f4d-5a8c01293876 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 09 00:42:56.908509 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:42:56.908509 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:42:56.908509 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:42:56.908509 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:42:56.908509 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:42:56.908509 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:42:56.908509 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:42:56.908509 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:42:56.908509 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:42:56.908509 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:42:56.908509 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:42:56.908509 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:42:56.908509 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:42:56.908509 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:42:56.908509 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:42:56.909443 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:42:56.909443 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:42:56.909443 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:42:56.909443 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:42:56.909443 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:42:56.909443 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:42:56.909443 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:42:56.909443 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:42:56.909443 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:42:56.909443 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:42:56.909443 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:42:56.909443 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:42:56.909443 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:42:56.909443 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:42:56.909443 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:42:56.909443 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:42:56.910241 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:42:56.910241 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:42:56.910241 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:42:56.910241 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:42:56.910241 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:42:56.910241 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:42:56.910241 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:42:56.910241 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:42:56.910241 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:42:56.910241 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:42:56.910241 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 09 00:42:56.910241 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:42:56.910241 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 245/500] 10.222.0.33 () {56 vars in 1102 bytes} [Sun Oct 9 00:42:56 2022] GET /identity/v3/roles/service => generated 85 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:42:56.910876 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a99f04e-b98a-41b5-9f4d-5a8c01293876 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:56.933742 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7bf50e2-8512-4c69-81b4-1c09ad0e72a4 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:56.967263 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-290da02b-5500-4c91-8a43-b374930e9e71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:56.967920 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-290da02b-5500-4c91-8a43-b374930e9e71 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:56.968203 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-290da02b-5500-4c91-8a43-b374930e9e71 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:56.968476 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-290da02b-5500-4c91-8a43-b374930e9e71 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:56.975349 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-290da02b-5500-4c91-8a43-b374930e9e71 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:56.976937 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-290da02b-5500-4c91-8a43-b374930e9e71 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:56.989688 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-290da02b-5500-4c91-8a43-b374930e9e71 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Oct 09 00:42:56.989688 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:42:56.989688 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:42:56.989688 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:42:56.989688 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:42:56.989688 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:42:56.989688 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:42:56.989688 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:42:56.989688 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:42:56.989688 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:42:56.989688 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:42:56.989688 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:42:56.989688 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 09 00:42:56.989688 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 09 00:42:56.989688 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 09 00:42:56.989688 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 09 00:42:56.990614 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:42:56.990614 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:42:56.990614 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 09 00:42:56.990614 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 09 00:42:56.990614 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 09 00:42:56.990614 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 09 00:42:56.990614 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:42:56.990614 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:42:56.990614 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:42:56.990614 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:42:56.990614 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:42:56.990614 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:42:56.990614 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:42:56.990614 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:42:56.990614 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:42:56.990614 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:42:56.990614 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:42:56.990614 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:42:56.991511 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:42:56.991511 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:42:56.991511 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 09 00:42:56.991511 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 09 00:42:56.991511 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 09 00:42:56.991511 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 09 00:42:56.991511 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 09 00:42:56.991511 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 09 00:42:56.991511 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Oct 09 00:42:56.991511 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:42:56.991511 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 256/501] 10.222.0.33 () {56 vars in 1099 bytes} [Sun Oct 9 00:42:56 2022] GET /identity/v3/users/glance => generated 84 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:42:56.992331 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b37ed51c-8f1e-4932-ad94-6f5d13021d47 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:56.992331 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b37ed51c-8f1e-4932-ad94-6f5d13021d47 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:56.992446 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b37ed51c-8f1e-4932-ad94-6f5d13021d47 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:57.010237 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-b37ed51c-8f1e-4932-ad94-6f5d13021d47 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:42:57.011422 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-b37ed51c-8f1e-4932-ad94-6f5d13021d47 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:42:57.011585 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a99f04e-b98a-41b5-9f4d-5a8c01293876 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:57.019890 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:57.041408 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:57.048962 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2ea27fa-b2c3-45ba-bd0c-fcf9781a34ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:57.049566 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ea27fa-b2c3-45ba-bd0c-fcf9781a34ab None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:57.049867 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ea27fa-b2c3-45ba-bd0c-fcf9781a34ab None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:57.050122 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ea27fa-b2c3-45ba-bd0c-fcf9781a34ab None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:57.078972 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 257/502] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:42:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 09 00:42:57.080308 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b37ed51c-8f1e-4932-ad94-6f5d13021d47 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:57.122361 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2ea27fa-b2c3-45ba-bd0c-fcf9781a34ab None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:57.124269 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2ea27fa-b2c3-45ba-bd0c-fcf9781a34ab None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:57.184046 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b37ed51c-8f1e-4932-ad94-6f5d13021d47 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:57.218857 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b69e7cc4-b375-41fd-ae60-707750f4f9d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:57.219918 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b69e7cc4-b375-41fd-ae60-707750f4f9d6 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:57.220203 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b69e7cc4-b375-41fd-ae60-707750f4f9d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:57.220777 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b69e7cc4-b375-41fd-ae60-707750f4f9d6 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:57.234580 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 246/503] 10.222.0.33 () {58 vars in 1282 bytes} [Sun Oct 9 00:42:56 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 320 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 09 00:42:57.234580 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2ea27fa-b2c3-45ba-bd0c-fcf9781a34ab None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:57.237991 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b69e7cc4-b375-41fd-ae60-707750f4f9d6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:42:57.238940 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:57.245234 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-b69e7cc4-b375-41fd-ae60-707750f4f9d6 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 09 00:42:57.245234 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:42:57.245234 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:42:57.245234 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:42:57.245234 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:42:57.245234 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:42:57.245234 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:42:57.245234 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:42:57.245234 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:42:57.245234 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:42:57.245234 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:42:57.245234 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:42:57.245234 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:42:57.245234 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:42:57.245234 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:42:57.245234 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:42:57.246078 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:42:57.246078 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:42:57.246078 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:42:57.246078 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:42:57.246078 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:42:57.246078 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:42:57.246078 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:42:57.246078 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:42:57.246078 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:42:57.246078 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:42:57.246078 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:42:57.246078 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:42:57.246078 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:42:57.246078 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:42:57.246078 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:42:57.246078 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:42:57.246856 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:42:57.246856 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:42:57.246856 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:42:57.246856 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:42:57.246856 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:42:57.246856 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:42:57.246856 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:42:57.246856 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:42:57.246856 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:42:57.246856 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:42:57.246856 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 09 00:42:57.246856 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:42:57.247448 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 258/504] 10.222.0.33 () {56 vars in 1102 bytes} [Sun Oct 9 00:42:57 2022] GET /identity/v3/roles/service => generated 85 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:42:57.248102 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b69e7cc4-b375-41fd-ae60-707750f4f9d6 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:57.280696 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:57.341088 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2ea27fa-b2c3-45ba-bd0c-fcf9781a34ab None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:57.355173 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b69e7cc4-b375-41fd-ae60-707750f4f9d6 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:57.374858 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0394089f-163b-4c29-bb29-2ddbd859569f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:57.375458 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0394089f-163b-4c29-bb29-2ddbd859569f None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:57.375776 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0394089f-163b-4c29-bb29-2ddbd859569f None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:57.376050 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0394089f-163b-4c29-bb29-2ddbd859569f None admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:57.376702 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0394089f-163b-4c29-bb29-2ddbd859569f None admin] RBAC: Adding query filter params (name=service) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:57.377003 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0394089f-163b-4c29-bb29-2ddbd859569f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:57.378351 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0394089f-163b-4c29-bb29-2ddbd859569f None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:57.386573 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 247/505] 10.222.0.33 () {56 vars in 1103 bytes} [Sun Oct 9 00:42:57 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:57.386573 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0394089f-163b-4c29-bb29-2ddbd859569f None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:57.394341 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4714dca4-3b16-4ffa-a37f-67897f3c2f48 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:57.394940 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4714dca4-3b16-4ffa-a37f-67897f3c2f48 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:57.395216 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4714dca4-3b16-4ffa-a37f-67897f3c2f48 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:57.395524 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4714dca4-3b16-4ffa-a37f-67897f3c2f48 None admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:57.396280 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4714dca4-3b16-4ffa-a37f-67897f3c2f48 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:57.396585 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4714dca4-3b16-4ffa-a37f-67897f3c2f48 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:57.398012 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4714dca4-3b16-4ffa-a37f-67897f3c2f48 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:57.415966 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 259/506] 10.222.0.33 () {56 vars in 1137 bytes} [Sun Oct 9 00:42:57 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:57.416759 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4714dca4-3b16-4ffa-a37f-67897f3c2f48 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:57.456759 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:57.489733 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0394089f-163b-4c29-bb29-2ddbd859569f None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:57.500290 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:57.515168 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4714dca4-3b16-4ffa-a37f-67897f3c2f48 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:57.522876 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35dfd84f-2973-4702-8180-62a94c566aa7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:57.523497 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35dfd84f-2973-4702-8180-62a94c566aa7 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:57.523816 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35dfd84f-2973-4702-8180-62a94c566aa7 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:57.524093 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35dfd84f-2973-4702-8180-62a94c566aa7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:57.534298 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35dfd84f-2973-4702-8180-62a94c566aa7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:57.535772 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35dfd84f-2973-4702-8180-62a94c566aa7 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:57.548177 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbdb0610-fc8e-468e-afd5-fe4fa70af925 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:57.548772 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbdb0610-fc8e-468e-afd5-fe4fa70af925 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:57.549053 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbdb0610-fc8e-468e-afd5-fe4fa70af925 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:57.549328 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbdb0610-fc8e-468e-afd5-fe4fa70af925 None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:57.549977 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbdb0610-fc8e-468e-afd5-fe4fa70af925 None admin] RBAC: Adding query filter params (name=service) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:57.550081 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-35dfd84f-2973-4702-8180-62a94c566aa7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:42:57.550081 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:42:57.550081 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:42:57.550081 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:42:57.550081 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:42:57.550081 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:42:57.550081 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:42:57.550081 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:42:57.550081 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:42:57.550081 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:42:57.550081 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:42:57.550081 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:42:57.550081 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:42:57.550081 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:42:57.550081 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:42:57.550081 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:42:57.551059 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:42:57.551059 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:42:57.551059 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:42:57.551059 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:42:57.551059 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:42:57.551059 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:42:57.551059 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:42:57.551059 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:42:57.551059 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:42:57.551059 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:42:57.551059 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:42:57.551059 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:42:57.551059 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:42:57.551059 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:42:57.551059 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:42:57.551059 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:42:57.551059 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:42:57.552003 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:42:57.552003 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:42:57.552003 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:42:57.552158 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbdb0610-fc8e-468e-afd5-fe4fa70af925 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:57.552223 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 248/507] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:42:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:42:57.552286 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbdb0610-fc8e-468e-afd5-fe4fa70af925 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:57.552362 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35dfd84f-2973-4702-8180-62a94c566aa7 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:57.563246 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 260/508] 10.222.0.33 () {56 vars in 1103 bytes} [Sun Oct 9 00:42:57 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:57.564066 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbdb0610-fc8e-468e-afd5-fe4fa70af925 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:57.651782 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35dfd84f-2973-4702-8180-62a94c566aa7 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:57.664447 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbdb0610-fc8e-468e-afd5-fe4fa70af925 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:57.671500 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:57.686824 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4004ec0f-0226-4312-a923-90dba2a09518 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:57.687444 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4004ec0f-0226-4312-a923-90dba2a09518 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:57.687752 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4004ec0f-0226-4312-a923-90dba2a09518 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:57.688033 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4004ec0f-0226-4312-a923-90dba2a09518 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:57.698183 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c785106-c24b-4b35-a269-7352f02195ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:57.698316 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4004ec0f-0226-4312-a923-90dba2a09518 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:57.698795 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c785106-c24b-4b35-a269-7352f02195ff None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:57.699073 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c785106-c24b-4b35-a269-7352f02195ff None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:57.699345 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c785106-c24b-4b35-a269-7352f02195ff None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:57.699794 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4004ec0f-0226-4312-a923-90dba2a09518 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:57.709526 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c785106-c24b-4b35-a269-7352f02195ff None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:57.710928 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c785106-c24b-4b35-a269-7352f02195ff None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:57.713815 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-4004ec0f-0226-4312-a923-90dba2a09518 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:42:57.713815 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:42:57.713815 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:42:57.713815 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:42:57.713815 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:42:57.713815 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:42:57.713815 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:42:57.713815 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:42:57.713815 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:42:57.713815 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:42:57.713815 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:42:57.713815 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:42:57.713815 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:42:57.713815 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:42:57.713815 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:42:57.713815 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:42:57.714834 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:42:57.714834 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:42:57.714834 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:42:57.714834 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:42:57.714834 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:42:57.714834 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:42:57.714834 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:42:57.714834 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:42:57.714834 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:42:57.714834 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:42:57.714834 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:42:57.714834 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:42:57.714834 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:42:57.714834 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:42:57.714834 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:42:57.714834 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:42:57.714834 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:42:57.715706 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:42:57.715706 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:42:57.715706 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:42:57.715706 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 249/509] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:42:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:42:57.716100 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4004ec0f-0226-4312-a923-90dba2a09518 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:57.718739 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:57.725851 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-6c785106-c24b-4b35-a269-7352f02195ff None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:42:57.725851 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:42:57.725851 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:42:57.725851 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:42:57.725851 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:42:57.725851 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:42:57.725851 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:42:57.725851 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:42:57.725851 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:42:57.725851 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:42:57.725851 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:42:57.725851 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:42:57.725851 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:42:57.725851 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:42:57.725851 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:42:57.725851 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:42:57.727437 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:42:57.727437 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:42:57.727437 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:42:57.727437 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:42:57.727437 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:42:57.727437 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:42:57.727437 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:42:57.727437 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:42:57.727437 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:42:57.727437 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:42:57.727437 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:42:57.727437 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:42:57.727437 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:42:57.727437 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:42:57.727437 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:42:57.727437 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:42:57.727437 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:42:57.728332 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:42:57.728332 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:42:57.728332 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:42:57.728332 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 261/510] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:42:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:42:57.728536 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c785106-c24b-4b35-a269-7352f02195ff None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:57.821950 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4004ec0f-0226-4312-a923-90dba2a09518 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:57.825967 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c785106-c24b-4b35-a269-7352f02195ff None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:57.854732 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-006731bc-944a-4e61-aeca-bee942f0bece None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:57.855330 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-006731bc-944a-4e61-aeca-bee942f0bece None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:57.855615 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-006731bc-944a-4e61-aeca-bee942f0bece None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:57.855905 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-006731bc-944a-4e61-aeca-bee942f0bece None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:57.858894 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d4fbc6b-2486-42bc-9309-367bd5873644 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:57.859486 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d4fbc6b-2486-42bc-9309-367bd5873644 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:57.859797 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d4fbc6b-2486-42bc-9309-367bd5873644 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:57.860066 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d4fbc6b-2486-42bc-9309-367bd5873644 None admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:57.860693 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d4fbc6b-2486-42bc-9309-367bd5873644 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:57.860994 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d4fbc6b-2486-42bc-9309-367bd5873644 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:57.862368 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d4fbc6b-2486-42bc-9309-367bd5873644 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:57.874189 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 262/511] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:42:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:57.875001 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d4fbc6b-2486-42bc-9309-367bd5873644 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:57.902662 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:57.930221 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-006731bc-944a-4e61-aeca-bee942f0bece None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:57.931751 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-006731bc-944a-4e61-aeca-bee942f0bece None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:57.934352 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:57.974943 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d4fbc6b-2486-42bc-9309-367bd5873644 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:58.008599 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-288e4319-1da1-417e-94e2-ccbc3be520f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:58.009222 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288e4319-1da1-417e-94e2-ccbc3be520f4 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:58.009502 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288e4319-1da1-417e-94e2-ccbc3be520f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:58.009771 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288e4319-1da1-417e-94e2-ccbc3be520f4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:58.019553 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-288e4319-1da1-417e-94e2-ccbc3be520f4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:58.021036 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-288e4319-1da1-417e-94e2-ccbc3be520f4 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:58.027276 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 250/512] 10.222.0.33 () {58 vars in 1282 bytes} [Sun Oct 9 00:42:57 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 09 00:42:58.028135 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-006731bc-944a-4e61-aeca-bee942f0bece None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:58.036004 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-288e4319-1da1-417e-94e2-ccbc3be520f4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:42:58.036004 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:42:58.036004 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:42:58.036004 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:42:58.036004 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:42:58.036004 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:42:58.036004 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:42:58.036004 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:42:58.036004 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:42:58.036004 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:42:58.036004 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:42:58.036004 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:42:58.036004 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:42:58.036004 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:42:58.036004 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:42:58.036004 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:42:58.036926 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:42:58.036926 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:42:58.036926 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:42:58.036926 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:42:58.036926 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:42:58.036926 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:42:58.036926 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:42:58.036926 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:42:58.036926 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:42:58.036926 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:42:58.036926 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:42:58.036926 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:42:58.036926 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:42:58.036926 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:42:58.036926 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:42:58.036926 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:42:58.036926 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:42:58.037751 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:42:58.037751 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:42:58.037751 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:42:58.037751 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 263/513] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:42:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:42:58.039836 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-288e4319-1da1-417e-94e2-ccbc3be520f4 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:58.098180 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:58.127805 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-006731bc-944a-4e61-aeca-bee942f0bece None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:58.142467 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-288e4319-1da1-417e-94e2-ccbc3be520f4 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:58.155227 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:58.161895 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6a83f10-45c1-427e-825d-f4d2758e4470 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:58.162501 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a83f10-45c1-427e-825d-f4d2758e4470 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:58.162781 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a83f10-45c1-427e-825d-f4d2758e4470 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:58.163317 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a83f10-45c1-427e-825d-f4d2758e4470 None admin] PATH_INFO: `/v3/domains` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:58.164581 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6a83f10-45c1-427e-825d-f4d2758e4470 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:58.164884 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6a83f10-45c1-427e-825d-f4d2758e4470 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:58.166254 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6a83f10-45c1-427e-825d-f4d2758e4470 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:58.177059 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e356c1a2-14a2-42b8-89e2-7f12e4d40933 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:58.177272 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 251/514] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:42:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:58.181931 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e356c1a2-14a2-42b8-89e2-7f12e4d40933 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:58.181931 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e356c1a2-14a2-42b8-89e2-7f12e4d40933 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:58.182112 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6a83f10-45c1-427e-825d-f4d2758e4470 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:58.182192 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e356c1a2-14a2-42b8-89e2-7f12e4d40933 None admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:58.182192 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e356c1a2-14a2-42b8-89e2-7f12e4d40933 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:58.182192 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e356c1a2-14a2-42b8-89e2-7f12e4d40933 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:58.182192 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e356c1a2-14a2-42b8-89e2-7f12e4d40933 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:58.192211 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 264/515] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:42:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:58.192993 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e356c1a2-14a2-42b8-89e2-7f12e4d40933 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:58.279493 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6a83f10-45c1-427e-825d-f4d2758e4470 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:58.291269 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e356c1a2-14a2-42b8-89e2-7f12e4d40933 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:58.307268 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:58.312573 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3c38186-e4e8-42c9-be20-a5948e1e2d9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:58.313204 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3c38186-e4e8-42c9-be20-a5948e1e2d9b None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:58.313690 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3c38186-e4e8-42c9-be20-a5948e1e2d9b None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:58.313961 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3c38186-e4e8-42c9-be20-a5948e1e2d9b None admin] PATH_INFO: `/v3/projects/service` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:58.321046 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3c38186-e4e8-42c9-be20-a5948e1e2d9b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:58.322485 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3c38186-e4e8-42c9-be20-a5948e1e2d9b None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:58.325161 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea4d4259-32fe-4688-99e9-080046b996dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:58.325775 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea4d4259-32fe-4688-99e9-080046b996dc None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:58.326053 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea4d4259-32fe-4688-99e9-080046b996dc None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:58.326325 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea4d4259-32fe-4688-99e9-080046b996dc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:58.333906 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-a3c38186-e4e8-42c9-be20-a5948e1e2d9b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 09 00:42:58.333906 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:42:58.333906 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:42:58.333906 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:42:58.333906 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:42:58.333906 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:42:58.333906 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:42:58.333906 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:42:58.333906 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:42:58.333906 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:42:58.333906 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:42:58.333906 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:42:58.333906 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 09 00:42:58.333906 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 09 00:42:58.333906 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 09 00:42:58.334704 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 09 00:42:58.334704 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:42:58.334704 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:42:58.334704 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:42:58.334704 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:42:58.334704 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:42:58.334704 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:42:58.334704 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:42:58.334704 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:42:58.334704 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:42:58.334704 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:42:58.334704 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:42:58.334704 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:42:58.334704 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:42:58.334704 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:42:58.334704 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:42:58.334704 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:42:58.335513 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 09 00:42:58.335513 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 09 00:42:58.335513 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 09 00:42:58.335513 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 09 00:42:58.335513 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 09 00:42:58.335513 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 09 00:42:58.335513 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 09 00:42:58.335513 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:42:58.335513 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 252/516] 10.222.0.33 () {56 vars in 1111 bytes} [Sun Oct 9 00:42:58 2022] GET /identity/v3/projects/service => generated 88 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:42:58.336986 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3c38186-e4e8-42c9-be20-a5948e1e2d9b None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:58.341502 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea4d4259-32fe-4688-99e9-080046b996dc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:58.342957 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea4d4259-32fe-4688-99e9-080046b996dc None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:58.357963 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-ea4d4259-32fe-4688-99e9-080046b996dc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:42:58.357963 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:42:58.357963 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:42:58.357963 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:42:58.357963 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:42:58.357963 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:42:58.357963 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:42:58.357963 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:42:58.357963 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:42:58.357963 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:42:58.357963 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:42:58.357963 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:42:58.357963 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:42:58.357963 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:42:58.357963 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:42:58.357963 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:42:58.358854 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:42:58.358854 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:42:58.358854 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:42:58.358854 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:42:58.358854 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:42:58.358854 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:42:58.358854 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:42:58.358854 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:42:58.358854 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:42:58.358854 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:42:58.358854 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:42:58.358854 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:42:58.358854 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:42:58.358854 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:42:58.358854 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:42:58.358854 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:42:58.358854 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:42:58.360196 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:42:58.360196 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:42:58.360196 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:42:58.360196 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 265/517] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:42:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:42:58.360196 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea4d4259-32fe-4688-99e9-080046b996dc None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:58.384964 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:58.437353 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3c38186-e4e8-42c9-be20-a5948e1e2d9b None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:58.458689 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea4d4259-32fe-4688-99e9-080046b996dc None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:58.470433 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27923554-b97a-45a1-a846-ce5391ee8716 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:58.471034 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27923554-b97a-45a1-a846-ce5391ee8716 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:58.471326 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27923554-b97a-45a1-a846-ce5391ee8716 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:58.471614 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27923554-b97a-45a1-a846-ce5391ee8716 None admin] PATH_INFO: `/v3/domains` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:58.472261 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27923554-b97a-45a1-a846-ce5391ee8716 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:58.472560 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27923554-b97a-45a1-a846-ce5391ee8716 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:58.473934 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27923554-b97a-45a1-a846-ce5391ee8716 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:58.486177 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 253/518] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:42:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:58.486871 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27923554-b97a-45a1-a846-ce5391ee8716 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:58.492783 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907a298f-93eb-44f5-b63b-0fccaafa68a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:58.493379 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907a298f-93eb-44f5-b63b-0fccaafa68a8 None admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:58.493658 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907a298f-93eb-44f5-b63b-0fccaafa68a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:58.493930 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907a298f-93eb-44f5-b63b-0fccaafa68a8 None admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:58.494563 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-907a298f-93eb-44f5-b63b-0fccaafa68a8 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:58.496029 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-907a298f-93eb-44f5-b63b-0fccaafa68a8 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:58.516817 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 266/519] 10.222.0.33 () {60 vars in 1132 bytes} [Sun Oct 9 00:42:58 2022] POST /identity/v3/users => generated 247 bytes in 157 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:42:58.517616 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907a298f-93eb-44f5-b63b-0fccaafa68a8 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:58.542488 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:58.598352 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27923554-b97a-45a1-a846-ce5391ee8716 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:58.625952 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:58.639192 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82c25a4f-9826-4014-91a5-431d182cbb01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:58.639870 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c25a4f-9826-4014-91a5-431d182cbb01 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:58.640151 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c25a4f-9826-4014-91a5-431d182cbb01 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:58.640423 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c25a4f-9826-4014-91a5-431d182cbb01 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:58.647689 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907a298f-93eb-44f5-b63b-0fccaafa68a8 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:58.648794 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82c25a4f-9826-4014-91a5-431d182cbb01 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:58.650412 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82c25a4f-9826-4014-91a5-431d182cbb01 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:58.662887 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-82c25a4f-9826-4014-91a5-431d182cbb01 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Oct 09 00:42:58.662887 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:42:58.662887 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:42:58.662887 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:42:58.662887 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:42:58.662887 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:42:58.662887 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:42:58.662887 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:42:58.662887 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:42:58.662887 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:42:58.662887 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:42:58.662887 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:42:58.662887 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 09 00:42:58.662887 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 09 00:42:58.662887 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 09 00:42:58.662887 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 09 00:42:58.664749 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:42:58.664749 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:42:58.664749 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 09 00:42:58.664749 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 09 00:42:58.664749 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 09 00:42:58.664749 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 09 00:42:58.664749 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:42:58.664749 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:42:58.664749 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:42:58.664749 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:42:58.664749 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:42:58.664749 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:42:58.664749 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:42:58.664749 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:42:58.664749 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:42:58.664749 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:42:58.664749 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:42:58.664749 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:42:58.665628 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:42:58.665628 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:42:58.665628 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 09 00:42:58.665628 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 09 00:42:58.665628 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 09 00:42:58.665628 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 09 00:42:58.665628 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 09 00:42:58.665628 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 09 00:42:58.665628 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Oct 09 00:42:58.665628 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:42:58.665628 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 254/520] 10.222.0.33 () {56 vars in 1099 bytes} [Sun Oct 9 00:42:58 2022] GET /identity/v3/users/cinder => generated 84 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:42:58.666856 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82c25a4f-9826-4014-91a5-431d182cbb01 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:58.690611 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e39b7c45-9485-42c9-a33c-0a590afe1c0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:58.691234 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e39b7c45-9485-42c9-a33c-0a590afe1c0d None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:58.691516 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e39b7c45-9485-42c9-a33c-0a590afe1c0d None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:58.691826 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e39b7c45-9485-42c9-a33c-0a590afe1c0d None admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:58.692528 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e39b7c45-9485-42c9-a33c-0a590afe1c0d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:58.692837 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e39b7c45-9485-42c9-a33c-0a590afe1c0d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:58.694292 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e39b7c45-9485-42c9-a33c-0a590afe1c0d None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:58.716206 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 267/521] 10.222.0.33 () {56 vars in 1148 bytes} [Sun Oct 9 00:42:58 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:58.717122 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e39b7c45-9485-42c9-a33c-0a590afe1c0d None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:58.781432 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82c25a4f-9826-4014-91a5-431d182cbb01 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:58.796633 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:58.818385 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-649d5a88-25b5-41cd-bd9d-07923c01b408 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:58.819025 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-649d5a88-25b5-41cd-bd9d-07923c01b408 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:58.819301 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-649d5a88-25b5-41cd-bd9d-07923c01b408 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:58.819635 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-649d5a88-25b5-41cd-bd9d-07923c01b408 None admin] PATH_INFO: `/v3/domains` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:58.820302 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-649d5a88-25b5-41cd-bd9d-07923c01b408 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:58.820596 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-649d5a88-25b5-41cd-bd9d-07923c01b408 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:58.822087 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-649d5a88-25b5-41cd-bd9d-07923c01b408 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:58.833916 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e39b7c45-9485-42c9-a33c-0a590afe1c0d None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:58.836247 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 255/522] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:42:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:58.838053 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-649d5a88-25b5-41cd-bd9d-07923c01b408 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:58.870506 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a73e2484-0611-45a9-a084-136394d34196 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:58.871125 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a73e2484-0611-45a9-a084-136394d34196 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:58.871381 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a73e2484-0611-45a9-a084-136394d34196 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:58.871675 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a73e2484-0611-45a9-a084-136394d34196 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:58.880297 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a73e2484-0611-45a9-a084-136394d34196 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:58.881887 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a73e2484-0611-45a9-a084-136394d34196 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:58.885555 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:58.895835 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-a73e2484-0611-45a9-a084-136394d34196 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Oct 09 00:42:58.895835 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:42:58.895835 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:42:58.895835 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:42:58.895835 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:42:58.895835 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:42:58.895835 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:42:58.895835 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:42:58.895835 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:42:58.895835 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:42:58.895835 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:42:58.895835 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:42:58.895835 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 09 00:42:58.895835 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 09 00:42:58.895835 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 09 00:42:58.895835 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 09 00:42:58.896939 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:42:58.896939 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:42:58.896939 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 09 00:42:58.896939 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 09 00:42:58.896939 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 09 00:42:58.896939 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 09 00:42:58.896939 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:42:58.896939 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:42:58.896939 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:42:58.896939 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:42:58.896939 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:42:58.896939 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:42:58.896939 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:42:58.896939 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:42:58.896939 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:42:58.896939 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:42:58.896939 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:42:58.896939 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:42:58.898684 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:42:58.898684 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:42:58.898684 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 09 00:42:58.898684 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 09 00:42:58.898684 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 09 00:42:58.898684 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 09 00:42:58.898684 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 09 00:42:58.898684 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 09 00:42:58.898684 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Oct 09 00:42:58.898684 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:42:58.905672 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 268/523] 10.222.0.33 () {56 vars in 1102 bytes} [Sun Oct 9 00:42:58 2022] GET /identity/v3/users/neutron => generated 85 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:42:58.905672 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a73e2484-0611-45a9-a084-136394d34196 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:58.953425 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-649d5a88-25b5-41cd-bd9d-07923c01b408 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:58.990522 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ded7fff-f168-46f2-b29f-e8e37e1f5cc7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:58.991131 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ded7fff-f168-46f2-b29f-e8e37e1f5cc7 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:58.991414 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ded7fff-f168-46f2-b29f-e8e37e1f5cc7 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:58.991955 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ded7fff-f168-46f2-b29f-e8e37e1f5cc7 None admin] PATH_INFO: `/v3/users` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:58.992480 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ded7fff-f168-46f2-b29f-e8e37e1f5cc7 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:58.992789 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ded7fff-f168-46f2-b29f-e8e37e1f5cc7 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:58.994205 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ded7fff-f168-46f2-b29f-e8e37e1f5cc7 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:59.006744 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a73e2484-0611-45a9-a084-136394d34196 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:59.032001 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 256/524] 10.222.0.33 () {56 vars in 1137 bytes} [Sun Oct 9 00:42:58 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:59.032840 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ded7fff-f168-46f2-b29f-e8e37e1f5cc7 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:59.048726 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49b6fea5-dcce-4f92-b81e-f098a6e0837c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:59.049254 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b6fea5-dcce-4f92-b81e-f098a6e0837c None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:59.049590 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b6fea5-dcce-4f92-b81e-f098a6e0837c None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:59.049866 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b6fea5-dcce-4f92-b81e-f098a6e0837c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:59.050628 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49b6fea5-dcce-4f92-b81e-f098a6e0837c None admin] RBAC: Adding query filter params (role.id=228232e130c5412a80d774978175d28b, scope.project.id=013c83476b1340fbb051061833c4a9c6, user.id=e0a5d95969e74157bb2d2a57207e1651) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:59.050938 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49b6fea5-dcce-4f92-b81e-f098a6e0837c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:59.052472 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49b6fea5-dcce-4f92-b81e-f098a6e0837c None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:59.067632 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 269/525] 10.222.0.33 () {56 vars in 1374 bytes} [Sun Oct 9 00:42:58 2022] GET /identity/v3/role_assignments?user.id=e0a5d95969e74157bb2d2a57207e1651&scope.project.id=013c83476b1340fbb051061833c4a9c6&role.id=228232e130c5412a80d774978175d28b => generated 259 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:59.067632 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49b6fea5-dcce-4f92-b81e-f098a6e0837c None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:59.087888 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:59.147006 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:59.177315 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ded7fff-f168-46f2-b29f-e8e37e1f5cc7 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:59.200898 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49b6fea5-dcce-4f92-b81e-f098a6e0837c None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:59.218759 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96d98910-b598-4f61-a20c-bf6b68901d08 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:59.219760 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d98910-b598-4f61-a20c-bf6b68901d08 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:59.220323 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d98910-b598-4f61-a20c-bf6b68901d08 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:59.220804 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d98910-b598-4f61-a20c-bf6b68901d08 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:59.229470 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96d98910-b598-4f61-a20c-bf6b68901d08 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:59.231294 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96d98910-b598-4f61-a20c-bf6b68901d08 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:59.240973 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8b996af-478d-40a5-8b23-0c59d8903c51 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:59.241591 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b996af-478d-40a5-8b23-0c59d8903c51 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:59.241929 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b996af-478d-40a5-8b23-0c59d8903c51 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:59.242710 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b996af-478d-40a5-8b23-0c59d8903c51 None admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:59.243442 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8b996af-478d-40a5-8b23-0c59d8903c51 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:59.243770 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8b996af-478d-40a5-8b23-0c59d8903c51 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:59.245221 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8b996af-478d-40a5-8b23-0c59d8903c51 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:59.246738 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-96d98910-b598-4f61-a20c-bf6b68901d08 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 09 00:42:59.246738 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:42:59.246738 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:42:59.246738 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:42:59.246738 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:42:59.246738 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:42:59.246738 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:42:59.246738 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:42:59.246738 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:42:59.246738 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:42:59.246738 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:42:59.246738 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:42:59.246738 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 09 00:42:59.246738 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 09 00:42:59.246738 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 09 00:42:59.247542 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 09 00:42:59.247542 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:42:59.247542 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:42:59.247542 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:42:59.247542 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:42:59.247542 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:42:59.247542 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:42:59.247542 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:42:59.247542 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:42:59.247542 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:42:59.247542 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:42:59.247542 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:42:59.247542 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:42:59.247542 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:42:59.247542 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:42:59.247542 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:42:59.247542 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:42:59.248505 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 09 00:42:59.248505 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 09 00:42:59.248505 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 09 00:42:59.248505 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 09 00:42:59.248505 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 09 00:42:59.248505 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 09 00:42:59.248505 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 09 00:42:59.248505 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:42:59.250771 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 257/526] 10.222.0.33 () {56 vars in 1111 bytes} [Sun Oct 9 00:42:59 2022] GET /identity/v3/projects/service => generated 88 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:42:59.251880 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96d98910-b598-4f61-a20c-bf6b68901d08 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:59.270529 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 270/527] 10.222.0.33 () {56 vars in 1139 bytes} [Sun Oct 9 00:42:59 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:59.271393 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8b996af-478d-40a5-8b23-0c59d8903c51 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:59.355801 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:59.369730 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96d98910-b598-4f61-a20c-bf6b68901d08 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:59.405487 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8b996af-478d-40a5-8b23-0c59d8903c51 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:59.415159 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b00de56d-5fea-43d0-ac26-fd08ed35cc12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:59.415800 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b00de56d-5fea-43d0-ac26-fd08ed35cc12 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:59.416079 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b00de56d-5fea-43d0-ac26-fd08ed35cc12 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:59.416352 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b00de56d-5fea-43d0-ac26-fd08ed35cc12 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:59.430641 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:59.445232 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a464706-a351-4665-ad7b-5f9a927b7d77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:59.446108 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a464706-a351-4665-ad7b-5f9a927b7d77 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:59.446527 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a464706-a351-4665-ad7b-5f9a927b7d77 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:59.446966 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a464706-a351-4665-ad7b-5f9a927b7d77 None admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:59.447908 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a464706-a351-4665-ad7b-5f9a927b7d77 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:59.448355 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a464706-a351-4665-ad7b-5f9a927b7d77 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:59.449992 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a464706-a351-4665-ad7b-5f9a927b7d77 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:59.469650 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 271/528] 10.222.0.33 () {56 vars in 1148 bytes} [Sun Oct 9 00:42:59 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:59.470870 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a464706-a351-4665-ad7b-5f9a927b7d77 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:59.505039 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b00de56d-5fea-43d0-ac26-fd08ed35cc12 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:59.506632 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b00de56d-5fea-43d0-ac26-fd08ed35cc12 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:59.575110 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a464706-a351-4665-ad7b-5f9a927b7d77 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:59.615285 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5752a54-2550-445a-9aa7-42d751e971bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:59.615852 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5752a54-2550-445a-9aa7-42d751e971bd None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:59.616123 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5752a54-2550-445a-9aa7-42d751e971bd None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:59.616412 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5752a54-2550-445a-9aa7-42d751e971bd None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:59.622341 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:59.634822 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 258/529] 10.222.0.33 () {58 vars in 1282 bytes} [Sun Oct 9 00:42:59 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 383 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 09 00:42:59.636136 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b00de56d-5fea-43d0-ac26-fd08ed35cc12 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:59.685568 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:42:59.710203 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5752a54-2550-445a-9aa7-42d751e971bd None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:59.712035 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5752a54-2550-445a-9aa7-42d751e971bd None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:59.754217 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b00de56d-5fea-43d0-ac26-fd08ed35cc12 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:42:59.800318 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ac98291-08b9-464d-93a1-5afb8cad7681 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:42:59.800926 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac98291-08b9-464d-93a1-5afb8cad7681 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:42:59.801203 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac98291-08b9-464d-93a1-5afb8cad7681 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:42:59.801475 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac98291-08b9-464d-93a1-5afb8cad7681 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:42:59.802201 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ac98291-08b9-464d-93a1-5afb8cad7681 None admin] RBAC: Adding query filter params (role.id=228232e130c5412a80d774978175d28b, scope.project.id=013c83476b1340fbb051061833c4a9c6, user.id=1f7a88094d73471787cc0a287adbd394) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:42:59.802509 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ac98291-08b9-464d-93a1-5afb8cad7681 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:42:59.803968 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ac98291-08b9-464d-93a1-5afb8cad7681 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:42:59.810552 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 259/530] 10.222.0.33 () {56 vars in 1374 bytes} [Sun Oct 9 00:42:59 2022] GET /identity/v3/role_assignments?user.id=1f7a88094d73471787cc0a287adbd394&scope.project.id=013c83476b1340fbb051061833c4a9c6&role.id=228232e130c5412a80d774978175d28b => generated 259 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:42:59.811416 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ac98291-08b9-464d-93a1-5afb8cad7681 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:59.840835 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 272/531] 10.222.0.33 () {58 vars in 1282 bytes} [Sun Oct 9 00:42:59 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 370 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 09 00:42:59.853246 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5752a54-2550-445a-9aa7-42d751e971bd None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:42:59.960297 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:00.013314 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:00.043284 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ac98291-08b9-464d-93a1-5afb8cad7681 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:00.049349 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5752a54-2550-445a-9aa7-42d751e971bd None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:00.097334 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01a4909b-7fd7-49de-b755-b6c47a7f5799 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:00.098241 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a4909b-7fd7-49de-b755-b6c47a7f5799 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:00.098774 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a4909b-7fd7-49de-b755-b6c47a7f5799 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:00.099237 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a4909b-7fd7-49de-b755-b6c47a7f5799 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:00.110891 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01a4909b-7fd7-49de-b755-b6c47a7f5799 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:00.112609 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01a4909b-7fd7-49de-b755-b6c47a7f5799 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:00.128369 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-01a4909b-7fd7-49de-b755-b6c47a7f5799 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:00.128369 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:00.128369 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:00.128369 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:00.128369 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:00.128369 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:00.128369 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:00.128369 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:00.128369 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:00.128369 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:00.128369 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:00.128369 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:00.128369 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:43:00.128369 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:43:00.128369 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:43:00.128369 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:43:00.129264 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:00.129264 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:00.129264 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:00.129264 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:00.129264 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:00.129264 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:00.129264 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:00.129264 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:00.129264 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:00.129264 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:00.129264 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:00.129264 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:00.129264 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:00.129264 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:00.129264 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:00.129264 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:00.129264 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:43:00.130092 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:43:00.130092 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:00.130092 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:00.132333 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 273/532] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:42:59 2022] GET /identity/v3/domains/Default => generated 87 bytes in 279 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 09 00:43:00.138035 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01a4909b-7fd7-49de-b755-b6c47a7f5799 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:00.141464 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92d7c7d0-b388-4708-b5e3-3f93d160a924 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:00.142124 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d7c7d0-b388-4708-b5e3-3f93d160a924 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:00.142393 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d7c7d0-b388-4708-b5e3-3f93d160a924 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:00.142656 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d7c7d0-b388-4708-b5e3-3f93d160a924 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:00.166220 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92d7c7d0-b388-4708-b5e3-3f93d160a924 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:00.167904 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92d7c7d0-b388-4708-b5e3-3f93d160a924 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:00.201029 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-92d7c7d0-b388-4708-b5e3-3f93d160a924 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:00.201029 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:00.201029 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:00.201029 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:00.201029 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:00.201029 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:00.201029 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:00.201029 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:00.201029 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:00.201029 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:00.201029 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:00.201029 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:00.201029 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:43:00.201029 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:43:00.201029 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:43:00.201029 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:43:00.201988 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:00.201988 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:00.201988 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:00.201988 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:00.201988 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:00.201988 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:00.201988 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:00.201988 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:00.201988 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:00.201988 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:00.201988 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:00.201988 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:00.201988 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:00.201988 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:00.201988 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:00.201988 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:00.201988 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:43:00.202799 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:43:00.202799 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:00.202799 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:00.202799 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 260/533] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:42:59 2022] GET /identity/v3/domains/Default => generated 87 bytes in 391 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:00.208900 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92d7c7d0-b388-4708-b5e3-3f93d160a924 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:00.250412 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01a4909b-7fd7-49de-b755-b6c47a7f5799 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:00.286893 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdb15c02-4742-400a-88a5-57c0f2e13a81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:00.287893 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb15c02-4742-400a-88a5-57c0f2e13a81 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:00.288640 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb15c02-4742-400a-88a5-57c0f2e13a81 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:00.289771 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb15c02-4742-400a-88a5-57c0f2e13a81 None admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:00.290683 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdb15c02-4742-400a-88a5-57c0f2e13a81 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:00.291172 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdb15c02-4742-400a-88a5-57c0f2e13a81 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:00.292773 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdb15c02-4742-400a-88a5-57c0f2e13a81 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:00.294399 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:00.306475 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 274/534] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:43:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:43:00.311870 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdb15c02-4742-400a-88a5-57c0f2e13a81 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:00.418787 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92d7c7d0-b388-4708-b5e3-3f93d160a924 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:00.421584 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdb15c02-4742-400a-88a5-57c0f2e13a81 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:00.426089 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:00.459962 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25daed7e-d41d-44f6-9db6-e51bc0e6718f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:00.461516 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4491d7e-5072-4c5e-8351-1a2db292bdfc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:00.461796 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25daed7e-d41d-44f6-9db6-e51bc0e6718f None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:00.462247 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4491d7e-5072-4c5e-8351-1a2db292bdfc None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:00.462439 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25daed7e-d41d-44f6-9db6-e51bc0e6718f None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:00.462599 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4491d7e-5072-4c5e-8351-1a2db292bdfc None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:00.462980 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4491d7e-5072-4c5e-8351-1a2db292bdfc None admin] PATH_INFO: `/v3/domains` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:00.463209 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25daed7e-d41d-44f6-9db6-e51bc0e6718f None admin] PATH_INFO: `/v3/projects/service` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:00.467851 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4491d7e-5072-4c5e-8351-1a2db292bdfc None admin] RBAC: Adding query filter params (name=Default) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:00.468195 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4491d7e-5072-4c5e-8351-1a2db292bdfc None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:00.469593 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4491d7e-5072-4c5e-8351-1a2db292bdfc None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:00.471215 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25daed7e-d41d-44f6-9db6-e51bc0e6718f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:00.472862 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25daed7e-d41d-44f6-9db6-e51bc0e6718f None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:00.486090 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-25daed7e-d41d-44f6-9db6-e51bc0e6718f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 09 00:43:00.486090 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:00.486090 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:00.486090 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:00.486090 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:00.486090 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:00.486090 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:00.486090 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:00.486090 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:00.486090 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:00.486090 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:00.486090 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:00.486090 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 09 00:43:00.486090 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 09 00:43:00.486090 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 09 00:43:00.486940 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 09 00:43:00.486940 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:00.486940 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:00.486940 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:00.486940 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:00.486940 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:00.486940 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:00.486940 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:00.486940 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:00.486940 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:00.486940 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:00.486940 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:00.486940 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:00.486940 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:00.486940 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:00.486940 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:00.486940 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:00.487783 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 09 00:43:00.487783 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 09 00:43:00.487783 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 09 00:43:00.487783 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 09 00:43:00.487783 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 09 00:43:00.487783 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 09 00:43:00.487783 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 09 00:43:00.487783 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:00.489924 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 275/535] 10.222.0.33 () {56 vars in 1111 bytes} [Sun Oct 9 00:43:00 2022] GET /identity/v3/projects/service => generated 88 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 09 00:43:00.497678 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 261/536] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:43:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 289 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:43:00.498649 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25daed7e-d41d-44f6-9db6-e51bc0e6718f None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:00.503396 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4491d7e-5072-4c5e-8351-1a2db292bdfc None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:00.563616 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:00.629205 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25daed7e-d41d-44f6-9db6-e51bc0e6718f None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:00.705909 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ff69d37-f43c-444b-9100-31d0430f4c11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:00.706534 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ff69d37-f43c-444b-9100-31d0430f4c11 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:00.706817 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ff69d37-f43c-444b-9100-31d0430f4c11 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:00.707094 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ff69d37-f43c-444b-9100-31d0430f4c11 None admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:00.707840 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ff69d37-f43c-444b-9100-31d0430f4c11 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:00.708147 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ff69d37-f43c-444b-9100-31d0430f4c11 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:00.708733 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4491d7e-5072-4c5e-8351-1a2db292bdfc None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:00.709636 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ff69d37-f43c-444b-9100-31d0430f4c11 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:00.737152 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 276/537] 10.222.0.33 () {56 vars in 1148 bytes} [Sun Oct 9 00:43:00 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:00.743119 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ff69d37-f43c-444b-9100-31d0430f4c11 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:00.766355 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5de70ce7-2d18-4e63-90c0-0fdbd9e0d547 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:00.767199 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de70ce7-2d18-4e63-90c0-0fdbd9e0d547 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:00.768280 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de70ce7-2d18-4e63-90c0-0fdbd9e0d547 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:00.768831 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de70ce7-2d18-4e63-90c0-0fdbd9e0d547 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:00.782404 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5de70ce7-2d18-4e63-90c0-0fdbd9e0d547 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:00.786863 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5de70ce7-2d18-4e63-90c0-0fdbd9e0d547 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:00.805877 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-5de70ce7-2d18-4e63-90c0-0fdbd9e0d547 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 09 00:43:00.805877 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:00.805877 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:00.805877 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:00.805877 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:00.805877 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:00.805877 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:00.805877 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:00.805877 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:00.805877 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:00.805877 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:00.805877 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:00.805877 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 09 00:43:00.805877 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 09 00:43:00.805877 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 09 00:43:00.806911 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 09 00:43:00.806911 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:00.806911 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:00.806911 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:00.806911 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:00.806911 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:00.806911 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:00.806911 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:00.806911 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:00.806911 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:00.806911 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:00.806911 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:00.806911 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:00.806911 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:00.806911 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:00.806911 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:00.806911 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:00.807956 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 09 00:43:00.807956 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 09 00:43:00.807956 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 09 00:43:00.807956 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 09 00:43:00.807956 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 09 00:43:00.807956 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 09 00:43:00.807956 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 09 00:43:00.807956 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:00.810433 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 262/538] 10.222.0.33 () {56 vars in 1111 bytes} [Sun Oct 9 00:43:00 2022] GET /identity/v3/projects/service => generated 88 bytes in 308 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 09 00:43:00.821787 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5de70ce7-2d18-4e63-90c0-0fdbd9e0d547 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:00.859902 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:00.889702 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ff69d37-f43c-444b-9100-31d0430f4c11 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:00.905891 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:00.928204 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66c276b7-d7ca-4825-813d-2d5858fbf0d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:00.928792 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c276b7-d7ca-4825-813d-2d5858fbf0d4 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:00.929067 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c276b7-d7ca-4825-813d-2d5858fbf0d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:00.929366 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c276b7-d7ca-4825-813d-2d5858fbf0d4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:00.930082 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66c276b7-d7ca-4825-813d-2d5858fbf0d4 None admin] RBAC: Adding query filter params (role.id=228232e130c5412a80d774978175d28b, scope.project.id=013c83476b1340fbb051061833c4a9c6, user.id=7d2eabf9865847ceb6e44d55126078a8) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:00.930390 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66c276b7-d7ca-4825-813d-2d5858fbf0d4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:00.931920 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66c276b7-d7ca-4825-813d-2d5858fbf0d4 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:00.937580 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 277/539] 10.222.0.33 () {56 vars in 1374 bytes} [Sun Oct 9 00:43:00 2022] GET /identity/v3/role_assignments?user.id=7d2eabf9865847ceb6e44d55126078a8&scope.project.id=013c83476b1340fbb051061833c4a9c6&role.id=228232e130c5412a80d774978175d28b => generated 259 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:43:00.938808 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e866877e-8b22-41ce-88ac-3b55565dcd79 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:00.939091 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e866877e-8b22-41ce-88ac-3b55565dcd79 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:00.939353 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e866877e-8b22-41ce-88ac-3b55565dcd79 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:00.944857 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 278/540] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:43:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:43:00.946463 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21d974b-b869-4338-98d0-891fdd091ff9 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:00.946746 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21d974b-b869-4338-98d0-891fdd091ff9 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:00.947017 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21d974b-b869-4338-98d0-891fdd091ff9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:00.958318 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5de70ce7-2d18-4e63-90c0-0fdbd9e0d547 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:00.966465 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-e21d974b-b869-4338-98d0-891fdd091ff9 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:43:00.967681 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-e21d974b-b869-4338-98d0-891fdd091ff9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:43:00.999301 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3386fa19-461e-41d1-ad39-64a2aafbe66b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:01.000801 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3386fa19-461e-41d1-ad39-64a2aafbe66b None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:01.001333 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3386fa19-461e-41d1-ad39-64a2aafbe66b None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:01.001806 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3386fa19-461e-41d1-ad39-64a2aafbe66b None admin] PATH_INFO: `/v3/projects` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:01.002845 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3386fa19-461e-41d1-ad39-64a2aafbe66b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:01.003459 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3386fa19-461e-41d1-ad39-64a2aafbe66b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:01.005240 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3386fa19-461e-41d1-ad39-64a2aafbe66b None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:01.026079 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 263/541] 10.222.0.33 () {56 vars in 1148 bytes} [Sun Oct 9 00:43:00 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:01.032548 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3386fa19-461e-41d1-ad39-64a2aafbe66b None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:01.041124 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 279/542] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:43:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 09 00:43:01.153836 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:01.170984 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3386fa19-461e-41d1-ad39-64a2aafbe66b None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:01.207793 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78b0570d-f7a5-4f22-92d9-44ddc39ee45f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:01.208406 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78b0570d-f7a5-4f22-92d9-44ddc39ee45f None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:01.208688 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78b0570d-f7a5-4f22-92d9-44ddc39ee45f None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:01.208964 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78b0570d-f7a5-4f22-92d9-44ddc39ee45f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:01.209692 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78b0570d-f7a5-4f22-92d9-44ddc39ee45f None admin] RBAC: Adding query filter params (role.id=228232e130c5412a80d774978175d28b, scope.project.id=013c83476b1340fbb051061833c4a9c6, user.id=f3f48335668a4e8f934ac11ff66cd308) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:01.210000 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78b0570d-f7a5-4f22-92d9-44ddc39ee45f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:01.211460 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78b0570d-f7a5-4f22-92d9-44ddc39ee45f None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:01.217622 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 264/543] 10.222.0.33 () {56 vars in 1374 bytes} [Sun Oct 9 00:43:01 2022] GET /identity/v3/role_assignments?user.id=f3f48335668a4e8f934ac11ff66cd308&scope.project.id=013c83476b1340fbb051061833c4a9c6&role.id=228232e130c5412a80d774978175d28b => generated 259 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:43:01.435818 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e21d974b-b869-4338-98d0-891fdd091ff9 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:01.522002 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6fcea4d-8d98-4b7e-8510-db5d40281f32 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:01.522323 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6fcea4d-8d98-4b7e-8510-db5d40281f32 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:01.522558 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6fcea4d-8d98-4b7e-8510-db5d40281f32 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:01.524694 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 265/544] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:43:01 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:43:01.531658 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980aaa66-2d4f-4409-aca1-aa9950ba7696 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:01.531947 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980aaa66-2d4f-4409-aca1-aa9950ba7696 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:01.532227 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980aaa66-2d4f-4409-aca1-aa9950ba7696 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:01.554084 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e21d974b-b869-4338-98d0-891fdd091ff9 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:01.564569 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:01.572629 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-980aaa66-2d4f-4409-aca1-aa9950ba7696 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:43:01.573846 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-980aaa66-2d4f-4409-aca1-aa9950ba7696 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:43:01.594568 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-038bf401-d9dd-4adb-b1db-ce1257d65174 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:01.595198 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-038bf401-d9dd-4adb-b1db-ce1257d65174 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:01.595477 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-038bf401-d9dd-4adb-b1db-ce1257d65174 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:01.595771 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-038bf401-d9dd-4adb-b1db-ce1257d65174 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:01.607706 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-038bf401-d9dd-4adb-b1db-ce1257d65174 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:43:01.613130 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-038bf401-d9dd-4adb-b1db-ce1257d65174 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 09 00:43:01.613130 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:01.613130 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:01.613130 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:01.613130 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:01.613130 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:01.613130 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:01.613130 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:01.613130 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:01.613130 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:01.613130 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:01.613130 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:01.613130 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:43:01.613130 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:43:01.613130 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:43:01.613130 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:43:01.613987 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:43:01.613987 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:43:01.613987 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:43:01.613987 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:43:01.613987 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:01.613987 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:01.613987 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:01.613987 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:01.613987 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:01.613987 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:01.613987 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:01.613987 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:01.613987 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:01.613987 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:01.613987 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:01.613987 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:01.615723 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:01.615723 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:01.615723 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:01.615723 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:01.615723 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:43:01.615723 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:43:01.615723 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:43:01.615723 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:43:01.615723 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:43:01.615723 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:43:01.615723 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 09 00:43:01.615723 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:01.615723 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 280/545] 10.222.0.33 () {56 vars in 1102 bytes} [Sun Oct 9 00:43:01 2022] GET /identity/v3/roles/service => generated 85 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:01.621448 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-038bf401-d9dd-4adb-b1db-ce1257d65174 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:01.738680 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 266/546] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:43:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 207 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 09 00:43:01.751492 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-038bf401-d9dd-4adb-b1db-ce1257d65174 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:01.804897 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62abc5e0-7d05-4b61-aa47-cea20590c3f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:01.805516 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62abc5e0-7d05-4b61-aa47-cea20590c3f6 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:01.805799 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62abc5e0-7d05-4b61-aa47-cea20590c3f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:01.806073 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62abc5e0-7d05-4b61-aa47-cea20590c3f6 None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:01.806736 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62abc5e0-7d05-4b61-aa47-cea20590c3f6 None admin] RBAC: Adding query filter params (name=service) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:01.807035 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62abc5e0-7d05-4b61-aa47-cea20590c3f6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:01.808453 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62abc5e0-7d05-4b61-aa47-cea20590c3f6 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:01.817431 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 281/547] 10.222.0.33 () {56 vars in 1103 bytes} [Sun Oct 9 00:43:01 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:43:01.823420 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-980aaa66-2d4f-4409-aca1-aa9950ba7696 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:01.921510 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:01.940813 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-980aaa66-2d4f-4409-aca1-aa9950ba7696 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:01.976832 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1e7ec82-12e4-48f4-bb7b-723717589772 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:01.977466 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e7ec82-12e4-48f4-bb7b-723717589772 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:01.977745 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e7ec82-12e4-48f4-bb7b-723717589772 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:01.978014 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e7ec82-12e4-48f4-bb7b-723717589772 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:01.988391 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1e7ec82-12e4-48f4-bb7b-723717589772 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:01.989901 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1e7ec82-12e4-48f4-bb7b-723717589772 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:02.004568 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-a1e7ec82-12e4-48f4-bb7b-723717589772 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:02.004568 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:02.004568 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:02.004568 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:02.004568 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:02.004568 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:02.004568 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:02.004568 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:02.004568 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:02.004568 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:02.004568 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:02.004568 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:02.004568 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:43:02.004568 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:43:02.004568 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:43:02.004568 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:43:02.007296 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:02.007296 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:02.007296 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:02.007296 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:02.007296 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:02.007296 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:02.007296 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:02.007296 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:02.007296 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:02.007296 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:02.007296 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:02.007296 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:02.007296 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:02.007296 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:02.007296 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:02.007296 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:02.007296 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:43:02.009982 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:43:02.009982 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:02.009982 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:02.009982 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 267/548] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:43:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:02.012066 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62abc5e0-7d05-4b61-aa47-cea20590c3f6 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:02.061666 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:02.161961 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62abc5e0-7d05-4b61-aa47-cea20590c3f6 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:02.217501 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9d73ecd-e172-415a-a5b7-a979ab701d8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:02.218392 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d73ecd-e172-415a-a5b7-a979ab701d8f None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:02.218889 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d73ecd-e172-415a-a5b7-a979ab701d8f None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:02.219357 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d73ecd-e172-415a-a5b7-a979ab701d8f None admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:02.220184 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9d73ecd-e172-415a-a5b7-a979ab701d8f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:02.220647 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9d73ecd-e172-415a-a5b7-a979ab701d8f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:02.222175 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9d73ecd-e172-415a-a5b7-a979ab701d8f None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:02.238609 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 282/549] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:43:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:02.244520 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1e7ec82-12e4-48f4-bb7b-723717589772 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:02.290854 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9d73ecd-e172-415a-a5b7-a979ab701d8f None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:02.395690 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1e7ec82-12e4-48f4-bb7b-723717589772 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:02.404394 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:02.418212 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9d73ecd-e172-415a-a5b7-a979ab701d8f None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:02.442398 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:02.449798 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d932bef8-329f-43fd-bc41-155b22d1fdf4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:02.450526 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d932bef8-329f-43fd-bc41-155b22d1fdf4 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:02.450811 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d932bef8-329f-43fd-bc41-155b22d1fdf4 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:02.451084 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d932bef8-329f-43fd-bc41-155b22d1fdf4 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:02.457516 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f23867b-f6a7-4225-89df-bdf3eb1eefd2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:02.458383 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f23867b-f6a7-4225-89df-bdf3eb1eefd2 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:02.459267 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f23867b-f6a7-4225-89df-bdf3eb1eefd2 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:02.459667 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d932bef8-329f-43fd-bc41-155b22d1fdf4 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:02.460012 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f23867b-f6a7-4225-89df-bdf3eb1eefd2 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:02.461657 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d932bef8-329f-43fd-bc41-155b22d1fdf4 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:02.472043 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f23867b-f6a7-4225-89df-bdf3eb1eefd2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:43:02.474737 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-d932bef8-329f-43fd-bc41-155b22d1fdf4 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 09 00:43:02.474737 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:02.474737 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:02.474737 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:02.474737 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:02.474737 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:02.474737 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:02.474737 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:02.474737 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:02.474737 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:02.474737 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:02.474737 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:02.474737 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 09 00:43:02.474737 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 09 00:43:02.474737 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 09 00:43:02.474737 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 09 00:43:02.475923 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:02.475923 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:02.475923 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 09 00:43:02.475923 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 09 00:43:02.475923 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 09 00:43:02.475923 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 09 00:43:02.475923 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:02.475923 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:02.475923 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:02.475923 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:02.475923 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:02.475923 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:02.475923 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:02.475923 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:02.475923 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:02.475923 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:02.475923 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:02.475923 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:02.477788 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:02.477788 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:02.477788 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 09 00:43:02.477788 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 09 00:43:02.477788 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 09 00:43:02.477788 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 09 00:43:02.477788 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 09 00:43:02.477788 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 09 00:43:02.477788 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 09 00:43:02.477788 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:02.477788 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 268/550] 10.222.0.33 () {56 vars in 1096 bytes} [Sun Oct 9 00:43:02 2022] GET /identity/v3/users/swift => generated 83 bytes in 232 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:02.478784 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf7f021-efef-4f6e-a2c9-64fa0f76e138 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:02.479078 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf7f021-efef-4f6e-a2c9-64fa0f76e138 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:02.479349 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf7f021-efef-4f6e-a2c9-64fa0f76e138 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:02.482222 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-3f23867b-f6a7-4225-89df-bdf3eb1eefd2 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 09 00:43:02.482222 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:02.482222 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:02.482222 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:02.482222 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:02.482222 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:02.482222 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:02.482222 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:02.482222 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:02.482222 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:02.482222 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:02.482222 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:02.482222 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:43:02.482222 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:43:02.482222 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:43:02.482222 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:43:02.483124 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:43:02.483124 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:43:02.483124 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:43:02.483124 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:43:02.483124 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:02.483124 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:02.483124 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:02.483124 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:02.483124 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:02.483124 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:02.483124 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:02.483124 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:02.483124 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:02.483124 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:02.483124 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:02.483124 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:02.484635 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:02.484635 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:02.484635 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:02.484635 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:02.484635 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:43:02.484635 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:43:02.484635 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:43:02.484635 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:43:02.484635 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:43:02.484635 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:43:02.484635 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 09 00:43:02.484635 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:02.485276 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 269/551] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:43:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:43:02.485512 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cf7f021-efef-4f6e-a2c9-64fa0f76e138 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:02.491585 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 283/552] 10.222.0.33 () {56 vars in 1102 bytes} [Sun Oct 9 00:43:02 2022] GET /identity/v3/roles/service => generated 85 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 09 00:43:02.493196 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c3b6988-dcf4-4b49-ac02-5943befe87b5 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:02.493689 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c3b6988-dcf4-4b49-ac02-5943befe87b5 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:02.494226 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c3b6988-dcf4-4b49-ac02-5943befe87b5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:02.515132 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-9c3b6988-dcf4-4b49-ac02-5943befe87b5 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:43:02.516731 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-9c3b6988-dcf4-4b49-ac02-5943befe87b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:43:02.594900 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 284/553] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:43:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 09 00:43:02.595853 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c3b6988-dcf4-4b49-ac02-5943befe87b5 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:02.640553 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cf7f021-efef-4f6e-a2c9-64fa0f76e138 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:02.705746 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2124471-332d-4182-a52f-f7f8bf946adb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:02.707145 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2124471-332d-4182-a52f-f7f8bf946adb None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:02.707440 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2124471-332d-4182-a52f-f7f8bf946adb None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:02.707784 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2124471-332d-4182-a52f-f7f8bf946adb None admin] PATH_INFO: `/v3/users` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:02.708508 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2124471-332d-4182-a52f-f7f8bf946adb None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:02.708814 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2124471-332d-4182-a52f-f7f8bf946adb None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:02.710248 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2124471-332d-4182-a52f-f7f8bf946adb None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:02.727847 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:02.746214 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 270/554] 10.222.0.33 () {56 vars in 1135 bytes} [Sun Oct 9 00:43:02 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 261 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:02.752291 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2124471-332d-4182-a52f-f7f8bf946adb None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:02.767815 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c3b6988-dcf4-4b49-ac02-5943befe87b5 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:02.814000 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6359b8e8-8390-4c8f-8610-fd39b89f85ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:02.814611 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6359b8e8-8390-4c8f-8610-fd39b89f85ed None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:02.814890 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6359b8e8-8390-4c8f-8610-fd39b89f85ed None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:02.815161 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6359b8e8-8390-4c8f-8610-fd39b89f85ed None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:02.815843 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6359b8e8-8390-4c8f-8610-fd39b89f85ed None admin] RBAC: Adding query filter params (name=service) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:02.816141 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6359b8e8-8390-4c8f-8610-fd39b89f85ed None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:02.817668 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6359b8e8-8390-4c8f-8610-fd39b89f85ed None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:02.827091 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 285/555] 10.222.0.33 () {56 vars in 1103 bytes} [Sun Oct 9 00:43:02 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:02.832491 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6359b8e8-8390-4c8f-8610-fd39b89f85ed None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:02.857618 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:02.942640 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2124471-332d-4182-a52f-f7f8bf946adb None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:02.984143 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6359b8e8-8390-4c8f-8610-fd39b89f85ed None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:02.990886 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5475d484-abe8-4844-85ea-d1ed14d690c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:02.992235 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5475d484-abe8-4844-85ea-d1ed14d690c8 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:02.992757 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5475d484-abe8-4844-85ea-d1ed14d690c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:02.993247 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5475d484-abe8-4844-85ea-d1ed14d690c8 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:03.019318 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-058828d9-11ca-4577-a7b5-6f73818064a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:03.019903 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-058828d9-11ca-4577-a7b5-6f73818064a5 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:03.020609 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-058828d9-11ca-4577-a7b5-6f73818064a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:03.020609 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-058828d9-11ca-4577-a7b5-6f73818064a5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:03.031744 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-058828d9-11ca-4577-a7b5-6f73818064a5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:03.033320 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-058828d9-11ca-4577-a7b5-6f73818064a5 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:03.035079 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:03.051515 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-058828d9-11ca-4577-a7b5-6f73818064a5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:03.051515 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:03.051515 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:03.051515 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:03.051515 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:03.051515 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:03.051515 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:03.051515 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:03.051515 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:03.051515 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:03.051515 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:03.051515 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:03.051515 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:43:03.051515 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:43:03.051515 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:43:03.051515 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:43:03.052621 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:03.052621 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:03.052621 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:03.052621 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:03.052621 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:03.052621 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:03.052621 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:03.052621 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:03.052621 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:03.052621 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:03.052621 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:03.052621 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:03.052621 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:03.052621 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:03.052621 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:03.052621 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:03.052621 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:43:03.053989 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:43:03.053989 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:03.053989 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:03.053989 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 286/556] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:43:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 09 00:43:03.053989 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-058828d9-11ca-4577-a7b5-6f73818064a5 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:03.087320 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5475d484-abe8-4844-85ea-d1ed14d690c8 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:03.088819 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5475d484-abe8-4844-85ea-d1ed14d690c8 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:03.150733 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:03.169075 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-058828d9-11ca-4577-a7b5-6f73818064a5 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:03.204653 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e409d97-1d21-4cbc-9c3b-ba43bc361c79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:03.205286 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e409d97-1d21-4cbc-9c3b-ba43bc361c79 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:03.205567 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e409d97-1d21-4cbc-9c3b-ba43bc361c79 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:03.205838 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e409d97-1d21-4cbc-9c3b-ba43bc361c79 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:03.214971 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 271/557] 10.222.0.33 () {58 vars in 1282 bytes} [Sun Oct 9 00:43:02 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 463 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 09 00:43:03.216517 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5475d484-abe8-4844-85ea-d1ed14d690c8 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:03.216821 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e409d97-1d21-4cbc-9c3b-ba43bc361c79 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:43:03.222067 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-4e409d97-1d21-4cbc-9c3b-ba43bc361c79 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 09 00:43:03.222067 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:03.222067 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:03.222067 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:03.222067 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:03.222067 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:03.222067 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:03.222067 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:03.222067 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:03.222067 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:03.222067 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:03.222067 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:03.222067 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:43:03.222067 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:43:03.222067 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:43:03.222067 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:43:03.223841 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:43:03.223841 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:43:03.223841 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:43:03.223841 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:43:03.223841 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:03.223841 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:03.223841 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:03.223841 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:03.223841 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:03.223841 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:03.223841 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:03.223841 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:03.223841 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:03.223841 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:03.223841 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:03.223841 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:03.224921 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:03.224921 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:03.224921 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:03.224921 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:03.224921 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:43:03.224921 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:43:03.224921 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:43:03.224921 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:43:03.224921 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:43:03.224921 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:43:03.224921 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 09 00:43:03.224921 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:03.224921 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 287/558] 10.222.0.33 () {56 vars in 1102 bytes} [Sun Oct 9 00:43:03 2022] GET /identity/v3/roles/service => generated 85 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:03.228585 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e409d97-1d21-4cbc-9c3b-ba43bc361c79 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:03.285729 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:03.335158 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e409d97-1d21-4cbc-9c3b-ba43bc361c79 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:03.337234 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5475d484-abe8-4844-85ea-d1ed14d690c8 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:03.370672 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-984f5132-7b0c-44a0-89a9-d2359f9b44ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:03.371557 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-984f5132-7b0c-44a0-89a9-d2359f9b44ac None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:03.372016 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-984f5132-7b0c-44a0-89a9-d2359f9b44ac None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:03.372430 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-984f5132-7b0c-44a0-89a9-d2359f9b44ac None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:03.377780 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2124a11e-7626-4b5c-aa1d-f744ce40c109 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:03.378384 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2124a11e-7626-4b5c-aa1d-f744ce40c109 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:03.378667 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2124a11e-7626-4b5c-aa1d-f744ce40c109 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:03.378940 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2124a11e-7626-4b5c-aa1d-f744ce40c109 None admin] PATH_INFO: `/v3/domains` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:03.379577 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2124a11e-7626-4b5c-aa1d-f744ce40c109 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:03.379923 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2124a11e-7626-4b5c-aa1d-f744ce40c109 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:03.381702 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2124a11e-7626-4b5c-aa1d-f744ce40c109 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:03.385509 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-984f5132-7b0c-44a0-89a9-d2359f9b44ac None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:03.387158 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-984f5132-7b0c-44a0-89a9-d2359f9b44ac None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:03.395033 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 272/559] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:43:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:03.395953 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2124a11e-7626-4b5c-aa1d-f744ce40c109 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:03.402845 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-984f5132-7b0c-44a0-89a9-d2359f9b44ac None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:03.402845 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:03.402845 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:03.402845 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:03.402845 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:03.402845 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:03.402845 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:03.402845 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:03.402845 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:03.402845 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:03.402845 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:03.402845 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:03.402845 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:43:03.402845 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:43:03.402845 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:43:03.402845 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:43:03.403821 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:03.403821 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:03.403821 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:03.403821 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:03.403821 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:03.403821 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:03.403821 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:03.403821 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:03.403821 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:03.403821 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:03.403821 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:03.403821 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:03.403821 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:03.403821 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:03.403821 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:03.403821 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:03.403821 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:43:03.404676 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:43:03.404676 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:03.404676 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:03.406758 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 288/560] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:43:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:03.407827 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-984f5132-7b0c-44a0-89a9-d2359f9b44ac None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:03.408712 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:03.518726 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2124a11e-7626-4b5c-aa1d-f744ce40c109 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:03.520837 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-984f5132-7b0c-44a0-89a9-d2359f9b44ac None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:03.528522 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:03.560559 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e465a97d-f699-467f-ba86-696b2160f2a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:03.561131 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e465a97d-f699-467f-ba86-696b2160f2a0 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:03.561417 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e465a97d-f699-467f-ba86-696b2160f2a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:03.561697 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e465a97d-f699-467f-ba86-696b2160f2a0 None admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:03.562374 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e465a97d-f699-467f-ba86-696b2160f2a0 None admin] RBAC: Adding query filter params (name=service) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:03.562682 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e465a97d-f699-467f-ba86-696b2160f2a0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:03.564134 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e465a97d-f699-467f-ba86-696b2160f2a0 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:03.565904 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3702029f-6a29-4d37-ae35-2a37c045b01d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:03.566823 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3702029f-6a29-4d37-ae35-2a37c045b01d None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:03.567329 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3702029f-6a29-4d37-ae35-2a37c045b01d None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:03.567856 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3702029f-6a29-4d37-ae35-2a37c045b01d None admin] PATH_INFO: `/v3/users/nova` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:03.574562 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 273/561] 10.222.0.33 () {56 vars in 1103 bytes} [Sun Oct 9 00:43:03 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:03.575535 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e465a97d-f699-467f-ba86-696b2160f2a0 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:03.578465 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3702029f-6a29-4d37-ae35-2a37c045b01d None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:03.580652 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3702029f-6a29-4d37-ae35-2a37c045b01d None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:03.594608 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-3702029f-6a29-4d37-ae35-2a37c045b01d None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 09 00:43:03.594608 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:03.594608 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:03.594608 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:03.594608 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:03.594608 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:03.594608 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:03.594608 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:03.594608 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:03.594608 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:03.594608 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:03.594608 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:03.594608 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 09 00:43:03.594608 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 09 00:43:03.594608 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 09 00:43:03.594608 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 09 00:43:03.595757 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:03.595757 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:03.595757 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 09 00:43:03.595757 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 09 00:43:03.595757 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 09 00:43:03.595757 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 09 00:43:03.595757 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:03.595757 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:03.595757 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:03.595757 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:03.595757 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:03.595757 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:03.595757 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:03.595757 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:03.595757 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:03.595757 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:03.595757 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:03.595757 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:03.596663 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:03.596663 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:03.596663 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 09 00:43:03.596663 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 09 00:43:03.596663 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 09 00:43:03.596663 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 09 00:43:03.596663 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 09 00:43:03.596663 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 09 00:43:03.596663 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 09 00:43:03.596663 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:03.598987 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 289/562] 10.222.0.33 () {56 vars in 1093 bytes} [Sun Oct 9 00:43:03 2022] GET /identity/v3/users/nova => generated 82 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:03.600151 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3702029f-6a29-4d37-ae35-2a37c045b01d None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:03.658014 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:03.690271 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e465a97d-f699-467f-ba86-696b2160f2a0 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:03.705647 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3702029f-6a29-4d37-ae35-2a37c045b01d None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:03.725387 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f21019f9-7581-4b85-a16d-03346cd57d1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:03.725988 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f21019f9-7581-4b85-a16d-03346cd57d1e None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:03.726298 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f21019f9-7581-4b85-a16d-03346cd57d1e None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:03.726576 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f21019f9-7581-4b85-a16d-03346cd57d1e None admin] PATH_INFO: `/v3/domains` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:03.727207 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f21019f9-7581-4b85-a16d-03346cd57d1e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:03.727510 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f21019f9-7581-4b85-a16d-03346cd57d1e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:03.728909 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f21019f9-7581-4b85-a16d-03346cd57d1e None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:03.739092 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2387f4d-f65a-42c0-ae95-17e9c45ee21b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:03.739742 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2387f4d-f65a-42c0-ae95-17e9c45ee21b None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:03.740021 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2387f4d-f65a-42c0-ae95-17e9c45ee21b None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:03.740293 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2387f4d-f65a-42c0-ae95-17e9c45ee21b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:03.740741 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 274/563] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:43:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:03.741531 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f21019f9-7581-4b85-a16d-03346cd57d1e None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:03.751904 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2387f4d-f65a-42c0-ae95-17e9c45ee21b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:03.753356 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2387f4d-f65a-42c0-ae95-17e9c45ee21b None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:03.768097 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-d2387f4d-f65a-42c0-ae95-17e9c45ee21b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:03.768097 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:03.768097 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:03.768097 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:03.768097 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:03.768097 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:03.768097 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:03.768097 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:03.768097 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:03.768097 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:03.768097 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:03.768097 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:03.768097 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:43:03.768097 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:43:03.768097 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:43:03.768097 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:43:03.769125 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:03.769125 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:03.769125 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:03.769125 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:03.769125 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:03.769125 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:03.769125 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:03.769125 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:03.769125 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:03.769125 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:03.769125 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:03.769125 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:03.769125 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:03.769125 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:03.769125 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:03.769125 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:03.769125 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:43:03.770171 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:43:03.770171 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:03.770171 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:03.770171 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 290/564] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:43:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:03.770676 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6add13c6-eb70-4ea4-85f5-ac2b31a5215e None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:03.770964 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6add13c6-eb70-4ea4-85f5-ac2b31a5215e None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:03.771239 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6add13c6-eb70-4ea4-85f5-ac2b31a5215e None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:03.772480 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 291/565] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:43:03 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:43:03.774181 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da883901-7dfb-42e8-a4a5-d6d462a3c2d9 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:03.774464 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da883901-7dfb-42e8-a4a5-d6d462a3c2d9 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:03.774740 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da883901-7dfb-42e8-a4a5-d6d462a3c2d9 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:03.776288 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 292/566] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:43:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:43:03.776987 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da883901-7dfb-42e8-a4a5-d6d462a3c2d9 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:03.796518 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:03.847138 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f21019f9-7581-4b85-a16d-03346cd57d1e None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:03.876327 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:03.879251 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da883901-7dfb-42e8-a4a5-d6d462a3c2d9 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:03.881899 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8b70260-1921-4feb-b90c-f0f20da2b467 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:03.883104 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b70260-1921-4feb-b90c-f0f20da2b467 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:03.883383 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b70260-1921-4feb-b90c-f0f20da2b467 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:03.883682 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b70260-1921-4feb-b90c-f0f20da2b467 None admin] PATH_INFO: `/v3/users` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:03.884395 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8b70260-1921-4feb-b90c-f0f20da2b467 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:03.884697 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8b70260-1921-4feb-b90c-f0f20da2b467 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:03.886119 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8b70260-1921-4feb-b90c-f0f20da2b467 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:03.905139 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 275/567] 10.222.0.33 () {56 vars in 1133 bytes} [Sun Oct 9 00:43:03 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:03.906121 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8b70260-1921-4feb-b90c-f0f20da2b467 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:03.917102 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecc25d8d-021d-439c-88f6-13cb436fdd40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:03.917728 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecc25d8d-021d-439c-88f6-13cb436fdd40 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:03.918008 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecc25d8d-021d-439c-88f6-13cb436fdd40 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:03.918285 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecc25d8d-021d-439c-88f6-13cb436fdd40 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:03.924804 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecc25d8d-021d-439c-88f6-13cb436fdd40 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:03.926224 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecc25d8d-021d-439c-88f6-13cb436fdd40 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:03.937692 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-ecc25d8d-021d-439c-88f6-13cb436fdd40 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 09 00:43:03.937692 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:03.937692 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:03.937692 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:03.937692 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:03.937692 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:03.937692 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:03.937692 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:03.937692 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:03.937692 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:03.937692 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:03.937692 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:03.937692 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 09 00:43:03.937692 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 09 00:43:03.937692 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 09 00:43:03.938736 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 09 00:43:03.938736 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:03.938736 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:03.938736 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:03.938736 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:03.938736 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:03.938736 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:03.938736 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:03.938736 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:03.938736 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:03.938736 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:03.938736 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:03.938736 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:03.938736 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:03.938736 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:03.938736 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:03.938736 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:03.939781 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 09 00:43:03.939781 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 09 00:43:03.939781 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 09 00:43:03.939781 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 09 00:43:03.939781 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 09 00:43:03.939781 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 09 00:43:03.939781 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 09 00:43:03.939781 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:03.939781 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 293/568] 10.222.0.33 () {56 vars in 1111 bytes} [Sun Oct 9 00:43:03 2022] GET /identity/v3/projects/service => generated 88 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:03.940278 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c04b7e2-695b-48fa-8ba5-b5bdd9672c3e None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:03.940508 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c04b7e2-695b-48fa-8ba5-b5bdd9672c3e None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:03.940780 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c04b7e2-695b-48fa-8ba5-b5bdd9672c3e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:03.958307 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-4c04b7e2-695b-48fa-8ba5-b5bdd9672c3e None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:43:03.959496 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-4c04b7e2-695b-48fa-8ba5-b5bdd9672c3e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:43:04.006322 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8b70260-1921-4feb-b90c-f0f20da2b467 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:04.023399 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 294/569] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:43:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 09 00:43:04.024782 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c8f0be3-e57f-49a4-84f7-397c24e183ab None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:04.025065 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c8f0be3-e57f-49a4-84f7-397c24e183ab None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:04.025335 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c8f0be3-e57f-49a4-84f7-397c24e183ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:04.042174 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-9c8f0be3-e57f-49a4-84f7-397c24e183ab None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:43:04.043333 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-9c8f0be3-e57f-49a4-84f7-397c24e183ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:43:04.045253 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:04.048379 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea31ccd5-5222-4182-b7f9-b1ab707638e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:04.048983 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea31ccd5-5222-4182-b7f9-b1ab707638e8 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:04.049260 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea31ccd5-5222-4182-b7f9-b1ab707638e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:04.049528 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea31ccd5-5222-4182-b7f9-b1ab707638e8 None admin] PATH_INFO: `/v3/domains` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:04.050153 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea31ccd5-5222-4182-b7f9-b1ab707638e8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:04.050448 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea31ccd5-5222-4182-b7f9-b1ab707638e8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:04.051886 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea31ccd5-5222-4182-b7f9-b1ab707638e8 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:04.064365 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 276/570] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:43:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:04.065155 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea31ccd5-5222-4182-b7f9-b1ab707638e8 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:04.121741 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:04.124581 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 295/571] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:43:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 09 00:43:04.125483 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c8f0be3-e57f-49a4-84f7-397c24e183ab None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:04.199494 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea31ccd5-5222-4182-b7f9-b1ab707638e8 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:04.227421 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c8f0be3-e57f-49a4-84f7-397c24e183ab None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:04.246816 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f18423fd-8263-4bff-b924-81303db70618 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:04.247795 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18423fd-8263-4bff-b924-81303db70618 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:04.248831 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18423fd-8263-4bff-b924-81303db70618 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:04.249309 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18423fd-8263-4bff-b924-81303db70618 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:04.263011 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6db5066c-fc68-401b-b6b6-bfa64ee73085 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:04.263700 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6db5066c-fc68-401b-b6b6-bfa64ee73085 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:04.264047 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6db5066c-fc68-401b-b6b6-bfa64ee73085 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:04.264388 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6db5066c-fc68-401b-b6b6-bfa64ee73085 None admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:04.264493 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f18423fd-8263-4bff-b924-81303db70618 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:04.265229 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6db5066c-fc68-401b-b6b6-bfa64ee73085 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:04.265592 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6db5066c-fc68-401b-b6b6-bfa64ee73085 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:04.266536 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f18423fd-8263-4bff-b924-81303db70618 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:04.267140 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6db5066c-fc68-401b-b6b6-bfa64ee73085 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:04.274789 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:04.280521 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 296/572] 10.222.0.33 () {56 vars in 1148 bytes} [Sun Oct 9 00:43:04 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:04.281557 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6db5066c-fc68-401b-b6b6-bfa64ee73085 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:04.285187 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-f18423fd-8263-4bff-b924-81303db70618 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:04.285187 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:04.285187 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:04.285187 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:04.285187 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:04.285187 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:04.285187 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:04.285187 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:04.285187 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:04.285187 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:04.285187 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:04.285187 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:04.285187 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:43:04.285187 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:43:04.285187 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:43:04.285187 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:43:04.286144 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:04.286144 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:04.286144 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:04.286144 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:04.286144 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:04.286144 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:04.286144 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:04.286144 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:04.286144 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:04.286144 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:04.286144 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:04.286144 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:04.286144 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:04.286144 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:04.286144 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:04.286144 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:04.286144 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:43:04.286994 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:43:04.286994 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:04.286994 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:04.289764 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 277/573] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:43:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 225 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:04.292398 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f18423fd-8263-4bff-b924-81303db70618 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:04.393525 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6db5066c-fc68-401b-b6b6-bfa64ee73085 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:04.397745 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f18423fd-8263-4bff-b924-81303db70618 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:04.399321 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:04.427843 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09e38839-bcdc-4ec5-bb27-5c624177b6f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:04.428462 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09e38839-bcdc-4ec5-bb27-5c624177b6f7 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:04.428743 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09e38839-bcdc-4ec5-bb27-5c624177b6f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:04.429014 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09e38839-bcdc-4ec5-bb27-5c624177b6f7 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:04.434072 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2c36e86-1b4a-4c79-8b97-d6b74dcacd55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:04.434649 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c36e86-1b4a-4c79-8b97-d6b74dcacd55 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:04.434929 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c36e86-1b4a-4c79-8b97-d6b74dcacd55 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:04.435200 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c36e86-1b4a-4c79-8b97-d6b74dcacd55 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:04.435960 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2c36e86-1b4a-4c79-8b97-d6b74dcacd55 None admin] RBAC: Adding query filter params (role.id=228232e130c5412a80d774978175d28b, scope.project.id=013c83476b1340fbb051061833c4a9c6, user.id=70e367e802894b08afd7d756d8d9ecff) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:04.436273 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2c36e86-1b4a-4c79-8b97-d6b74dcacd55 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:04.437692 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2c36e86-1b4a-4c79-8b97-d6b74dcacd55 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:04.438063 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09e38839-bcdc-4ec5-bb27-5c624177b6f7 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:04.439505 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09e38839-bcdc-4ec5-bb27-5c624177b6f7 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:04.442991 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 278/574] 10.222.0.33 () {56 vars in 1374 bytes} [Sun Oct 9 00:43:04 2022] GET /identity/v3/role_assignments?user.id=70e367e802894b08afd7d756d8d9ecff&scope.project.id=013c83476b1340fbb051061833c4a9c6&role.id=228232e130c5412a80d774978175d28b => generated 259 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:04.443811 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2c36e86-1b4a-4c79-8b97-d6b74dcacd55 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:04.451202 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-09e38839-bcdc-4ec5-bb27-5c624177b6f7 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Oct 09 00:43:04.451202 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:04.451202 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:04.451202 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:04.451202 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:04.451202 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:04.451202 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:04.451202 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:04.451202 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:04.451202 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:04.451202 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:04.451202 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:04.451202 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 09 00:43:04.451202 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 09 00:43:04.451202 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 09 00:43:04.451202 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 09 00:43:04.452262 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:04.452262 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:04.452262 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 09 00:43:04.452262 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 09 00:43:04.452262 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 09 00:43:04.452262 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 09 00:43:04.452262 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:04.452262 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:04.452262 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:04.452262 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:04.452262 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:04.452262 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:04.452262 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:04.452262 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:04.452262 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:04.452262 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:04.452262 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:04.452262 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:04.453179 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:04.453179 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:04.453179 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 09 00:43:04.453179 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 09 00:43:04.453179 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 09 00:43:04.453179 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 09 00:43:04.453179 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 09 00:43:04.453179 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 09 00:43:04.453179 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Oct 09 00:43:04.453179 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:04.453179 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 297/575] 10.222.0.33 () {56 vars in 1099 bytes} [Sun Oct 9 00:43:04 2022] GET /identity/v3/users/glance => generated 84 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:04.453749 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09e38839-bcdc-4ec5-bb27-5c624177b6f7 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:04.526077 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:04.567948 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2c36e86-1b4a-4c79-8b97-d6b74dcacd55 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:04.573566 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09e38839-bcdc-4ec5-bb27-5c624177b6f7 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:04.609346 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80499df0-e6e8-480e-8abe-681e1783f8b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:04.609973 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80499df0-e6e8-480e-8abe-681e1783f8b2 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:04.610255 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80499df0-e6e8-480e-8abe-681e1783f8b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:04.610536 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80499df0-e6e8-480e-8abe-681e1783f8b2 None admin] PATH_INFO: `/v3/domains` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:04.611186 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80499df0-e6e8-480e-8abe-681e1783f8b2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:04.611491 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80499df0-e6e8-480e-8abe-681e1783f8b2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:04.612918 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80499df0-e6e8-480e-8abe-681e1783f8b2 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:04.615889 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38361dee-6791-4dce-9f4d-49ca85722ad2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:04.616806 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38361dee-6791-4dce-9f4d-49ca85722ad2 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:04.617295 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38361dee-6791-4dce-9f4d-49ca85722ad2 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:04.617763 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38361dee-6791-4dce-9f4d-49ca85722ad2 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:04.625711 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 279/576] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:43:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:04.626338 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80499df0-e6e8-480e-8abe-681e1783f8b2 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:04.632192 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38361dee-6791-4dce-9f4d-49ca85722ad2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:43:04.637450 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-38361dee-6791-4dce-9f4d-49ca85722ad2 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 09 00:43:04.637450 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:04.637450 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:04.637450 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:04.637450 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:04.637450 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:04.637450 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:04.637450 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:04.637450 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:04.637450 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:04.637450 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:04.637450 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:04.637450 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:43:04.637450 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:43:04.637450 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:43:04.637450 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:43:04.638551 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:43:04.638551 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:43:04.638551 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:43:04.638551 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:43:04.638551 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:04.638551 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:04.638551 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:04.638551 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:04.638551 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:04.638551 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:04.638551 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:04.638551 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:04.638551 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:04.638551 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:04.638551 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:04.638551 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:04.639424 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:04.639424 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:04.639424 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:04.639424 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:04.639424 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:43:04.639424 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:43:04.639424 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:43:04.639424 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:43:04.639424 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:43:04.639424 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:43:04.639424 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 09 00:43:04.639424 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:04.639424 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 298/577] 10.222.0.33 () {56 vars in 1102 bytes} [Sun Oct 9 00:43:04 2022] GET /identity/v3/roles/service => generated 85 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:04.640202 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38361dee-6791-4dce-9f4d-49ca85722ad2 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:04.663221 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:04.741001 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80499df0-e6e8-480e-8abe-681e1783f8b2 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:04.748151 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38361dee-6791-4dce-9f4d-49ca85722ad2 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:04.766289 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:04.780796 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-798aeb6b-01f4-4423-b07b-5d7b6447637b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:04.781423 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-798aeb6b-01f4-4423-b07b-5d7b6447637b None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:04.781712 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-798aeb6b-01f4-4423-b07b-5d7b6447637b None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:04.781967 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-798aeb6b-01f4-4423-b07b-5d7b6447637b None admin] PATH_INFO: `/v3/roles/service` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:04.785439 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-799d1a11-bda2-40a3-bd09-0c30d1cd8e7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:04.786045 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799d1a11-bda2-40a3-bd09-0c30d1cd8e7a None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:04.786327 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799d1a11-bda2-40a3-bd09-0c30d1cd8e7a None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:04.786758 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799d1a11-bda2-40a3-bd09-0c30d1cd8e7a None admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:04.787644 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-799d1a11-bda2-40a3-bd09-0c30d1cd8e7a None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:04.787935 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-799d1a11-bda2-40a3-bd09-0c30d1cd8e7a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:04.790627 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-799d1a11-bda2-40a3-bd09-0c30d1cd8e7a None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:04.792946 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-798aeb6b-01f4-4423-b07b-5d7b6447637b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:43:04.798512 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-798aeb6b-01f4-4423-b07b-5d7b6447637b None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 09 00:43:04.798512 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:04.798512 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:04.798512 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:04.798512 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:04.798512 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:04.798512 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:04.798512 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:04.798512 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:04.798512 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:04.798512 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:04.798512 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:04.798512 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:43:04.798512 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:43:04.798512 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:43:04.798512 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:43:04.799371 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:43:04.799371 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:43:04.799371 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:43:04.799371 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:43:04.799371 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:04.799371 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:04.799371 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:04.799371 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:04.799371 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:04.799371 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:04.799371 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:04.799371 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:04.799371 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:04.799371 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:04.799371 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:04.799371 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:04.800632 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:04.800632 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:04.800632 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:04.800632 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:04.800632 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:43:04.800632 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:43:04.800632 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:43:04.800632 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:43:04.800632 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:43:04.800632 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:43:04.800632 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 09 00:43:04.800632 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:04.800632 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 280/578] 10.222.0.33 () {56 vars in 1102 bytes} [Sun Oct 9 00:43:04 2022] GET /identity/v3/roles/service => generated 85 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:04.800632 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-798aeb6b-01f4-4423-b07b-5d7b6447637b None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:04.814054 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 299/579] 10.222.0.33 () {56 vars in 1137 bytes} [Sun Oct 9 00:43:04 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:04.815269 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-799d1a11-bda2-40a3-bd09-0c30d1cd8e7a None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:04.893466 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:04.909642 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-798aeb6b-01f4-4423-b07b-5d7b6447637b None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:04.923401 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-799d1a11-bda2-40a3-bd09-0c30d1cd8e7a None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:04.950295 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41e8ab8d-b951-437d-95d3-2d5100635924 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:04.951184 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e8ab8d-b951-437d-95d3-2d5100635924 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:04.951464 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e8ab8d-b951-437d-95d3-2d5100635924 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:04.951786 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e8ab8d-b951-437d-95d3-2d5100635924 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:04.957301 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c604af27-380e-477b-97dd-09a066e111f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:04.957902 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c604af27-380e-477b-97dd-09a066e111f8 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:04.958181 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c604af27-380e-477b-97dd-09a066e111f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:04.958451 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c604af27-380e-477b-97dd-09a066e111f8 None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:04.959104 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c604af27-380e-477b-97dd-09a066e111f8 None admin] RBAC: Adding query filter params (name=service) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:04.959406 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c604af27-380e-477b-97dd-09a066e111f8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:04.959563 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41e8ab8d-b951-437d-95d3-2d5100635924 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:04.961370 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c604af27-380e-477b-97dd-09a066e111f8 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:04.961592 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41e8ab8d-b951-437d-95d3-2d5100635924 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:04.969877 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 300/580] 10.222.0.33 () {56 vars in 1103 bytes} [Sun Oct 9 00:43:04 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:04.970926 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c604af27-380e-477b-97dd-09a066e111f8 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:04.973120 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-41e8ab8d-b951-437d-95d3-2d5100635924 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 09 00:43:04.973120 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:04.973120 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:04.973120 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:04.973120 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:04.973120 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:04.973120 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:04.973120 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:04.973120 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:04.973120 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:04.973120 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:04.973120 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:04.973120 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 09 00:43:04.973120 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 09 00:43:04.973120 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 09 00:43:04.974070 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 09 00:43:04.974070 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:04.974070 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:04.974070 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:04.974070 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:04.974070 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:04.974070 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:04.974070 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:04.974070 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:04.974070 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:04.974070 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:04.974070 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:04.974070 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:04.974070 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:04.974070 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:04.974070 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:04.974070 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:04.974923 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 09 00:43:04.974923 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 09 00:43:04.974923 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 09 00:43:04.974923 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 09 00:43:04.974923 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 09 00:43:04.974923 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 09 00:43:04.974923 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 09 00:43:04.974923 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:04.974923 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 281/581] 10.222.0.33 () {56 vars in 1111 bytes} [Sun Oct 9 00:43:04 2022] GET /identity/v3/projects/service => generated 88 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:04.975398 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41e8ab8d-b951-437d-95d3-2d5100635924 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:05.019905 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:05.089356 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41e8ab8d-b951-437d-95d3-2d5100635924 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:05.096036 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c604af27-380e-477b-97dd-09a066e111f8 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:05.123914 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7127370-ef96-479e-a21a-51026fa9a8df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:05.124576 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7127370-ef96-479e-a21a-51026fa9a8df None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:05.124825 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7127370-ef96-479e-a21a-51026fa9a8df None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:05.125103 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7127370-ef96-479e-a21a-51026fa9a8df None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:05.130520 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0ed144b-da3c-4c26-b153-d127ab7dcb46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:05.131122 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ed144b-da3c-4c26-b153-d127ab7dcb46 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:05.131400 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ed144b-da3c-4c26-b153-d127ab7dcb46 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:05.131701 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ed144b-da3c-4c26-b153-d127ab7dcb46 None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:05.132950 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0ed144b-da3c-4c26-b153-d127ab7dcb46 None admin] RBAC: Adding query filter params (name=service) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:05.133254 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0ed144b-da3c-4c26-b153-d127ab7dcb46 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:05.134656 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0ed144b-da3c-4c26-b153-d127ab7dcb46 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:05.137156 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7127370-ef96-479e-a21a-51026fa9a8df None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:05.138592 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7127370-ef96-479e-a21a-51026fa9a8df None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:05.147640 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 301/582] 10.222.0.33 () {56 vars in 1103 bytes} [Sun Oct 9 00:43:04 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:05.147640 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0ed144b-da3c-4c26-b153-d127ab7dcb46 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:05.148718 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:05.158446 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-a7127370-ef96-479e-a21a-51026fa9a8df None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:05.158446 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:05.158446 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:05.158446 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:05.158446 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:05.158446 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:05.158446 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:05.158446 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:05.158446 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:05.158446 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:05.158446 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:05.158446 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:05.158446 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:43:05.158446 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:43:05.158446 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:43:05.158446 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:43:05.159359 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:05.159359 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:05.159359 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:05.159359 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:05.159359 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:05.159359 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:05.159359 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:05.159359 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:05.159359 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:05.159359 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:05.159359 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:05.159359 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:05.159359 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:05.159359 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:05.159359 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:05.159359 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:05.159359 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:43:05.160258 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:43:05.160258 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:05.160258 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:05.164103 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 282/583] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:43:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:05.165257 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7127370-ef96-479e-a21a-51026fa9a8df None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:05.251026 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:05.254090 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0ed144b-da3c-4c26-b153-d127ab7dcb46 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:05.273954 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7127370-ef96-479e-a21a-51026fa9a8df None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:05.291636 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ad22c85-5dbf-4946-a3d9-5a273e28e4b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:05.292466 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ad22c85-5dbf-4946-a3d9-5a273e28e4b8 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:05.292807 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ad22c85-5dbf-4946-a3d9-5a273e28e4b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:05.293138 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ad22c85-5dbf-4946-a3d9-5a273e28e4b8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:05.303569 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ad22c85-5dbf-4946-a3d9-5a273e28e4b8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:05.305712 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ad22c85-5dbf-4946-a3d9-5a273e28e4b8 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:05.309237 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3f57331-9574-48ea-b445-0a42457c4067 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:05.309863 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f57331-9574-48ea-b445-0a42457c4067 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:05.310199 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f57331-9574-48ea-b445-0a42457c4067 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:05.310478 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f57331-9574-48ea-b445-0a42457c4067 None admin] PATH_INFO: `/v3/projects` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:05.311206 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3f57331-9574-48ea-b445-0a42457c4067 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:05.311517 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3f57331-9574-48ea-b445-0a42457c4067 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:05.313742 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3f57331-9574-48ea-b445-0a42457c4067 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:05.322512 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-8ad22c85-5dbf-4946-a3d9-5a273e28e4b8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:05.322512 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:05.322512 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:05.322512 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:05.322512 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:05.322512 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:05.322512 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:05.322512 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:05.322512 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:05.322512 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:05.322512 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:05.322512 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:05.322512 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:43:05.322512 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:43:05.322512 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:43:05.322512 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:43:05.323839 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:05.323839 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:05.323839 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:05.323839 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:05.323839 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:05.323839 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:05.323839 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:05.323839 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:05.323839 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:05.323839 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:05.323839 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:05.323839 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:05.323839 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:05.323839 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:05.323839 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:05.323839 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:05.323839 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:43:05.324889 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:43:05.324889 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:05.324889 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:05.324889 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 302/584] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:43:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:05.326976 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ad22c85-5dbf-4946-a3d9-5a273e28e4b8 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:05.332146 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 283/585] 10.222.0.33 () {56 vars in 1148 bytes} [Sun Oct 9 00:43:05 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:05.332656 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3f57331-9574-48ea-b445-0a42457c4067 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:05.395343 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:05.434102 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ad22c85-5dbf-4946-a3d9-5a273e28e4b8 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:05.439114 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3f57331-9574-48ea-b445-0a42457c4067 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:05.469150 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-704ac496-1407-4f21-b617-1988b85c2870 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:05.469769 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-704ac496-1407-4f21-b617-1988b85c2870 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:05.470059 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-704ac496-1407-4f21-b617-1988b85c2870 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:05.470315 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-704ac496-1407-4f21-b617-1988b85c2870 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:05.476287 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ee1a2fb-f571-4bbe-ace6-0e7766694a55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:05.477177 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee1a2fb-f571-4bbe-ace6-0e7766694a55 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:05.477460 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee1a2fb-f571-4bbe-ace6-0e7766694a55 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:05.477728 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee1a2fb-f571-4bbe-ace6-0e7766694a55 None admin] PATH_INFO: `/v3/domains` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:05.478422 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ee1a2fb-f571-4bbe-ace6-0e7766694a55 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:05.478839 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ee1a2fb-f571-4bbe-ace6-0e7766694a55 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:05.480212 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ee1a2fb-f571-4bbe-ace6-0e7766694a55 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:05.480866 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-704ac496-1407-4f21-b617-1988b85c2870 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:05.482347 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-704ac496-1407-4f21-b617-1988b85c2870 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:05.497091 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 284/586] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:43:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:05.497091 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ee1a2fb-f571-4bbe-ace6-0e7766694a55 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:05.497358 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-704ac496-1407-4f21-b617-1988b85c2870 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:05.497358 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:05.497358 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:05.497358 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:05.497358 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:05.497358 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:05.497358 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:05.497358 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:05.497358 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:05.497358 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:05.497358 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:05.497358 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:05.497358 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:43:05.497358 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:43:05.497358 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:43:05.497358 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:43:05.498251 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:05.498251 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:05.498251 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:05.498251 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:05.498251 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:05.498251 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:05.498251 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:05.498251 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:05.498251 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:05.498251 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:05.498251 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:05.498251 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:05.498251 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:05.498251 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:05.498251 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:05.498251 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:05.498251 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:43:05.499124 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:43:05.499124 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:05.499124 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:05.499124 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 303/587] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:43:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:05.499124 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-704ac496-1407-4f21-b617-1988b85c2870 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:05.505321 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:05.608643 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-704ac496-1407-4f21-b617-1988b85c2870 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:05.610983 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ee1a2fb-f571-4bbe-ace6-0e7766694a55 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:05.628732 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:05.644064 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc8911a6-26a3-4e9f-971e-1635e2ef5f78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:05.644805 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc8911a6-26a3-4e9f-971e-1635e2ef5f78 None admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:05.645153 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc8911a6-26a3-4e9f-971e-1635e2ef5f78 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:05.645500 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc8911a6-26a3-4e9f-971e-1635e2ef5f78 None admin] PATH_INFO: `/v3/projects/013c83476b1340fbb051061833c4a9c6/users/e0a5d95969e74157bb2d2a57207e1651/roles/228232e130c5412a80d774978175d28b` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:05.649422 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-845eddfe-81a5-4d2d-98fc-ead21d1b5554 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:05.650094 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845eddfe-81a5-4d2d-98fc-ead21d1b5554 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:05.650440 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845eddfe-81a5-4d2d-98fc-ead21d1b5554 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:05.650779 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845eddfe-81a5-4d2d-98fc-ead21d1b5554 None admin] PATH_INFO: `/v3/domains` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:05.651486 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-845eddfe-81a5-4d2d-98fc-ead21d1b5554 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:05.651879 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-845eddfe-81a5-4d2d-98fc-ead21d1b5554 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:05.653376 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-845eddfe-81a5-4d2d-98fc-ead21d1b5554 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:05.660856 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc8911a6-26a3-4e9f-971e-1635e2ef5f78 None admin] RBAC: Authorizing `identity:create_grant(project_id=013c83476b1340fbb051061833c4a9c6, user_id=e0a5d95969e74157bb2d2a57207e1651, role_id=228232e130c5412a80d774978175d28b)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:05.662410 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc8911a6-26a3-4e9f-971e-1635e2ef5f78 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:05.668471 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 285/588] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:43:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:05.669461 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-845eddfe-81a5-4d2d-98fc-ead21d1b5554 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:05.687626 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 304/589] 10.222.0.33 () {58 vars in 1439 bytes} [Sun Oct 9 00:43:05 2022] PUT /identity/v3/projects/013c83476b1340fbb051061833c4a9c6/users/e0a5d95969e74157bb2d2a57207e1651/roles/228232e130c5412a80d774978175d28b => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:43:05.687626 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc8911a6-26a3-4e9f-971e-1635e2ef5f78 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:05.773937 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:05.809216 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-845eddfe-81a5-4d2d-98fc-ead21d1b5554 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:05.831322 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc8911a6-26a3-4e9f-971e-1635e2ef5f78 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:05.859519 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a7bf8d1-cada-4473-9dd1-f8815f206b13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:05.861118 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a7bf8d1-cada-4473-9dd1-f8815f206b13 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:05.861118 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a7bf8d1-cada-4473-9dd1-f8815f206b13 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:05.861118 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a7bf8d1-cada-4473-9dd1-f8815f206b13 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:05.872888 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a7bf8d1-cada-4473-9dd1-f8815f206b13 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:05.875069 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a7bf8d1-cada-4473-9dd1-f8815f206b13 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:05.878585 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0b731dd-b102-4124-bc74-f7105bbc2317 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:05.879191 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b731dd-b102-4124-bc74-f7105bbc2317 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:05.879468 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b731dd-b102-4124-bc74-f7105bbc2317 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:05.879859 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b731dd-b102-4124-bc74-f7105bbc2317 None admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:05.880640 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0b731dd-b102-4124-bc74-f7105bbc2317 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:05.880829 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0b731dd-b102-4124-bc74-f7105bbc2317 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:05.882127 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0b731dd-b102-4124-bc74-f7105bbc2317 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:05.887171 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-4a7bf8d1-cada-4473-9dd1-f8815f206b13 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 09 00:43:05.887171 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:05.887171 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:05.887171 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:05.887171 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:05.887171 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:05.887171 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:05.887171 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:05.887171 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:05.887171 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:05.887171 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:05.887171 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:05.887171 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 09 00:43:05.887171 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 09 00:43:05.887171 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 09 00:43:05.888102 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 09 00:43:05.888102 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:05.888102 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:05.888102 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:05.888102 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:05.888102 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:05.888102 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:05.888102 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:05.888102 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:05.888102 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:05.888102 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:05.888102 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:05.888102 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:05.888102 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:05.888102 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:05.888102 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:05.888102 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:05.888969 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 09 00:43:05.888969 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 09 00:43:05.888969 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 09 00:43:05.888969 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 09 00:43:05.888969 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 09 00:43:05.888969 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 09 00:43:05.888969 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 09 00:43:05.888969 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:05.888969 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 286/590] 10.222.0.33 () {56 vars in 1111 bytes} [Sun Oct 9 00:43:05 2022] GET /identity/v3/projects/service => generated 88 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:05.888969 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a7bf8d1-cada-4473-9dd1-f8815f206b13 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:05.895495 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 305/591] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:43:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:05.896243 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0b731dd-b102-4124-bc74-f7105bbc2317 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:05.939280 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:06.011469 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a7bf8d1-cada-4473-9dd1-f8815f206b13 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:06.029395 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0b731dd-b102-4124-bc74-f7105bbc2317 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:06.048440 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:06.053303 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44d74dce-db74-4c50-9b57-41ad19d427fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:06.053940 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d74dce-db74-4c50-9b57-41ad19d427fd None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:06.054248 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d74dce-db74-4c50-9b57-41ad19d427fd None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:06.054525 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d74dce-db74-4c50-9b57-41ad19d427fd None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:06.063285 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44d74dce-db74-4c50-9b57-41ad19d427fd None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:06.064764 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44d74dce-db74-4c50-9b57-41ad19d427fd None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:06.077579 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a12eaf26-8a23-4c3d-a174-ae6e5fff88a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:06.078505 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a12eaf26-8a23-4c3d-a174-ae6e5fff88a0 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:06.078968 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a12eaf26-8a23-4c3d-a174-ae6e5fff88a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:06.079420 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a12eaf26-8a23-4c3d-a174-ae6e5fff88a0 None admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:06.080314 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a12eaf26-8a23-4c3d-a174-ae6e5fff88a0 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:06.080822 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a12eaf26-8a23-4c3d-a174-ae6e5fff88a0 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:06.083184 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a12eaf26-8a23-4c3d-a174-ae6e5fff88a0 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:06.087403 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-44d74dce-db74-4c50-9b57-41ad19d427fd None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Oct 09 00:43:06.087403 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:06.087403 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:06.087403 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:06.087403 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:06.087403 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:06.087403 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:06.087403 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:06.087403 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:06.087403 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:06.087403 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:06.087403 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:06.087403 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 09 00:43:06.087403 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 09 00:43:06.087403 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 09 00:43:06.087403 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 09 00:43:06.088377 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:06.088377 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:06.088377 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 09 00:43:06.088377 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 09 00:43:06.088377 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 09 00:43:06.088377 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 09 00:43:06.088377 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:06.088377 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:06.088377 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:06.088377 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:06.088377 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:06.088377 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:06.088377 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:06.088377 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:06.088377 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:06.088377 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:06.088377 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:06.088377 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:06.089292 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:06.089292 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:06.089292 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 09 00:43:06.089292 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 09 00:43:06.089292 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 09 00:43:06.089292 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 09 00:43:06.089292 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 09 00:43:06.089292 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 09 00:43:06.089292 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Oct 09 00:43:06.089292 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:06.093743 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 287/592] 10.222.0.33 () {56 vars in 1102 bytes} [Sun Oct 9 00:43:05 2022] GET /identity/v3/users/neutron => generated 85 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:06.094744 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44d74dce-db74-4c50-9b57-41ad19d427fd None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:06.108883 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 306/593] 10.222.0.33 () {56 vars in 1148 bytes} [Sun Oct 9 00:43:05 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:06.108883 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a12eaf26-8a23-4c3d-a174-ae6e5fff88a0 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:06.214121 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a12eaf26-8a23-4c3d-a174-ae6e5fff88a0 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:06.224040 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:06.227585 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44d74dce-db74-4c50-9b57-41ad19d427fd None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:06.252650 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a143be3-b179-482d-be35-35d0c44c2f5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:06.253838 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a143be3-b179-482d-be35-35d0c44c2f5e None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:06.254268 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a143be3-b179-482d-be35-35d0c44c2f5e None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:06.254684 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a143be3-b179-482d-be35-35d0c44c2f5e None admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:06.255610 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a143be3-b179-482d-be35-35d0c44c2f5e None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:06.256059 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a143be3-b179-482d-be35-35d0c44c2f5e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:06.258324 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a143be3-b179-482d-be35-35d0c44c2f5e None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:06.265910 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-203c0a9b-41dd-4843-8e38-07174adcab32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:06.266541 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-203c0a9b-41dd-4843-8e38-07174adcab32 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:06.266821 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-203c0a9b-41dd-4843-8e38-07174adcab32 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:06.267192 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-203c0a9b-41dd-4843-8e38-07174adcab32 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:06.275492 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-203c0a9b-41dd-4843-8e38-07174adcab32 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:06.277106 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-203c0a9b-41dd-4843-8e38-07174adcab32 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:06.282119 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 307/594] 10.222.0.33 () {56 vars in 1139 bytes} [Sun Oct 9 00:43:06 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:06.283131 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a143be3-b179-482d-be35-35d0c44c2f5e None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:06.289571 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-203c0a9b-41dd-4843-8e38-07174adcab32 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Oct 09 00:43:06.289571 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:06.289571 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:06.289571 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:06.289571 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:06.289571 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:06.289571 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:06.289571 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:06.289571 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:06.289571 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:06.289571 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:06.289571 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:06.289571 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 09 00:43:06.289571 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 09 00:43:06.289571 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 09 00:43:06.289571 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 09 00:43:06.290710 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:06.290710 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:06.290710 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 09 00:43:06.290710 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 09 00:43:06.290710 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 09 00:43:06.290710 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 09 00:43:06.290710 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:06.290710 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:06.290710 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:06.290710 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:06.290710 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:06.290710 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:06.290710 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:06.290710 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:06.290710 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:06.290710 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:06.290710 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:06.290710 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:06.291792 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:06.291792 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:06.291792 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 09 00:43:06.291792 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 09 00:43:06.291792 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 09 00:43:06.291792 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 09 00:43:06.291792 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 09 00:43:06.291792 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 09 00:43:06.291792 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Oct 09 00:43:06.291792 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:06.291792 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 288/595] 10.222.0.33 () {56 vars in 1099 bytes} [Sun Oct 9 00:43:06 2022] GET /identity/v3/users/cinder => generated 84 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:06.291792 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-203c0a9b-41dd-4843-8e38-07174adcab32 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:06.318958 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:06.390891 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a143be3-b179-482d-be35-35d0c44c2f5e None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:06.404403 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-203c0a9b-41dd-4843-8e38-07174adcab32 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:06.426728 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b76cb67-d148-48c6-bf73-be9d0b472c0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:06.427641 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b76cb67-d148-48c6-bf73-be9d0b472c0b None admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:06.428094 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b76cb67-d148-48c6-bf73-be9d0b472c0b None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:06.428509 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b76cb67-d148-48c6-bf73-be9d0b472c0b None admin] PATH_INFO: `/v3/projects/013c83476b1340fbb051061833c4a9c6/users/1f7a88094d73471787cc0a287adbd394/roles/228232e130c5412a80d774978175d28b` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:06.443046 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79d3ef13-837f-4da1-968d-4161d86b843f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:06.443712 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d3ef13-837f-4da1-968d-4161d86b843f None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:06.443993 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d3ef13-837f-4da1-968d-4161d86b843f None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:06.444263 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d3ef13-837f-4da1-968d-4161d86b843f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:06.445950 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b76cb67-d148-48c6-bf73-be9d0b472c0b None admin] RBAC: Authorizing `identity:create_grant(project_id=013c83476b1340fbb051061833c4a9c6, user_id=1f7a88094d73471787cc0a287adbd394, role_id=228232e130c5412a80d774978175d28b)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:06.447544 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b76cb67-d148-48c6-bf73-be9d0b472c0b None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:06.456892 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79d3ef13-837f-4da1-968d-4161d86b843f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:06.458438 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79d3ef13-837f-4da1-968d-4161d86b843f None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:06.463353 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 308/596] 10.222.0.33 () {58 vars in 1439 bytes} [Sun Oct 9 00:43:06 2022] PUT /identity/v3/projects/013c83476b1340fbb051061833c4a9c6/users/1f7a88094d73471787cc0a287adbd394/roles/228232e130c5412a80d774978175d28b => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:43:06.464459 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b76cb67-d148-48c6-bf73-be9d0b472c0b None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:06.474718 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-79d3ef13-837f-4da1-968d-4161d86b843f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:06.474718 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:06.474718 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:06.474718 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:06.474718 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:06.474718 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:06.474718 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:06.474718 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:06.474718 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:06.474718 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:06.474718 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:06.474718 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:06.474718 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:43:06.474718 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:43:06.474718 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:43:06.474718 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:43:06.475938 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:06.475938 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:06.475938 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:06.475938 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:06.475938 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:06.475938 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:06.475938 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:06.475938 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:06.475938 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:06.475938 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:06.475938 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:06.475938 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:06.475938 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:06.475938 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:06.475938 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:06.475938 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:06.475938 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:43:06.476974 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:43:06.476974 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:06.476974 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:06.479936 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 289/597] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:43:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:06.481555 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79d3ef13-837f-4da1-968d-4161d86b843f None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:06.495264 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:06.572026 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:06.593645 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b76cb67-d148-48c6-bf73-be9d0b472c0b None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:06.603809 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79d3ef13-837f-4da1-968d-4161d86b843f None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:06.634412 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d380c68-f5e5-4df6-9b33-8d64b7987579 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:06.635089 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d380c68-f5e5-4df6-9b33-8d64b7987579 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:06.635375 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d380c68-f5e5-4df6-9b33-8d64b7987579 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:06.635670 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d380c68-f5e5-4df6-9b33-8d64b7987579 None admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:06.638729 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d380c68-f5e5-4df6-9b33-8d64b7987579 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:06.639209 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d380c68-f5e5-4df6-9b33-8d64b7987579 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:06.640903 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d380c68-f5e5-4df6-9b33-8d64b7987579 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:06.643322 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56b208f7-3402-41e0-aae8-08de0f9cb2b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:06.644305 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56b208f7-3402-41e0-aae8-08de0f9cb2b6 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:06.644765 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56b208f7-3402-41e0-aae8-08de0f9cb2b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:06.645193 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56b208f7-3402-41e0-aae8-08de0f9cb2b6 None admin] PATH_INFO: `/v3/domains` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:06.646024 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56b208f7-3402-41e0-aae8-08de0f9cb2b6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:06.646468 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56b208f7-3402-41e0-aae8-08de0f9cb2b6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:06.648628 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56b208f7-3402-41e0-aae8-08de0f9cb2b6 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:06.661608 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 309/598] 10.222.0.33 () {56 vars in 1137 bytes} [Sun Oct 9 00:43:06 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:06.662577 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 290/599] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:43:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:06.666749 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57117ab3-0ebb-44ec-9483-8775ab31a786 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:06.666749 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57117ab3-0ebb-44ec-9483-8775ab31a786 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:06.666749 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57117ab3-0ebb-44ec-9483-8775ab31a786 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:06.666749 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 310/600] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:43:06 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:43:06.668062 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56b208f7-3402-41e0-aae8-08de0f9cb2b6 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:06.672916 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57117ab3-0ebb-44ec-9483-8775ab31a786 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:06.747961 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:06.791448 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56b208f7-3402-41e0-aae8-08de0f9cb2b6 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:06.801199 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57117ab3-0ebb-44ec-9483-8775ab31a786 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:06.837928 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7faa7532-dfce-4b08-956e-4c777512e3c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:06.838792 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7faa7532-dfce-4b08-956e-4c777512e3c6 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:06.839483 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7faa7532-dfce-4b08-956e-4c777512e3c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:06.840378 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7faa7532-dfce-4b08-956e-4c777512e3c6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:06.845906 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f407aa9e-1377-4035-a06c-1cadc0707416 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:06.846601 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f407aa9e-1377-4035-a06c-1cadc0707416 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:06.846954 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f407aa9e-1377-4035-a06c-1cadc0707416 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:06.847291 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f407aa9e-1377-4035-a06c-1cadc0707416 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:06.853450 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7faa7532-dfce-4b08-956e-4c777512e3c6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:06.855459 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7faa7532-dfce-4b08-956e-4c777512e3c6 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:06.855920 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f407aa9e-1377-4035-a06c-1cadc0707416 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:06.857880 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f407aa9e-1377-4035-a06c-1cadc0707416 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:06.861295 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:06.873727 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-f407aa9e-1377-4035-a06c-1cadc0707416 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 09 00:43:06.873727 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:06.873727 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:06.873727 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:06.873727 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:06.873727 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:06.873727 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:06.873727 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:06.873727 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:06.873727 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:06.873727 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:06.873727 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:06.873727 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 09 00:43:06.873727 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 09 00:43:06.873727 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 09 00:43:06.874624 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 09 00:43:06.874624 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:06.874624 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:06.874624 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:06.874624 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:06.874624 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:06.874624 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:06.874624 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:06.874624 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:06.874624 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:06.874624 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:06.874624 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:06.874624 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:06.874624 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:06.874624 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:06.874624 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:06.874624 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:06.876209 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 09 00:43:06.876209 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 09 00:43:06.876209 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 09 00:43:06.876209 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 09 00:43:06.876209 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 09 00:43:06.876209 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 09 00:43:06.876209 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 09 00:43:06.876209 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:06.876209 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 311/601] 10.222.0.33 () {56 vars in 1111 bytes} [Sun Oct 9 00:43:06 2022] GET /identity/v3/projects/service => generated 88 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:06.880521 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a7d8725-1a3b-4d4d-983d-75ae7f46915d None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:06.880795 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-7faa7532-dfce-4b08-956e-4c777512e3c6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:06.880795 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:06.880795 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:06.880795 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:06.880795 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:06.880795 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:06.880795 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:06.880795 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:06.880795 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:06.880795 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:06.880795 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:06.880795 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:06.880795 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:43:06.880795 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:43:06.880795 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:43:06.880795 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:43:06.881638 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:06.881638 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:06.881638 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:06.881638 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:06.881638 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:06.881638 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:06.881638 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:06.881638 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:06.881638 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:06.881638 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:06.881638 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:06.881638 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:06.881638 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:06.881638 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:06.881638 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:06.881638 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:06.881638 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:43:06.883418 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:43:06.883418 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:06.883418 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:06.883418 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 291/602] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:43:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:06.883663 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7faa7532-dfce-4b08-956e-4c777512e3c6 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:06.884216 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a7d8725-1a3b-4d4d-983d-75ae7f46915d None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:06.886138 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a7d8725-1a3b-4d4d-983d-75ae7f46915d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:06.908630 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-0a7d8725-1a3b-4d4d-983d-75ae7f46915d None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:43:06.910699 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-0a7d8725-1a3b-4d4d-983d-75ae7f46915d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:43:07.005561 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 312/603] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:43:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 09 00:43:07.005561 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a7d8725-1a3b-4d4d-983d-75ae7f46915d None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:07.024102 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7faa7532-dfce-4b08-956e-4c777512e3c6 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:07.072044 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:07.100497 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff05455a-f2a1-4027-91e2-b1dfa4f15318 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:07.101073 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff05455a-f2a1-4027-91e2-b1dfa4f15318 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:07.101353 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff05455a-f2a1-4027-91e2-b1dfa4f15318 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:07.101629 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff05455a-f2a1-4027-91e2-b1dfa4f15318 None admin] PATH_INFO: `/v3/projects` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:07.102323 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff05455a-f2a1-4027-91e2-b1dfa4f15318 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:07.102629 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff05455a-f2a1-4027-91e2-b1dfa4f15318 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:07.104158 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff05455a-f2a1-4027-91e2-b1dfa4f15318 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:07.126266 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 292/604] 10.222.0.33 () {56 vars in 1148 bytes} [Sun Oct 9 00:43:06 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:07.132011 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff05455a-f2a1-4027-91e2-b1dfa4f15318 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:07.201205 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a7d8725-1a3b-4d4d-983d-75ae7f46915d None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:07.260085 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:07.269615 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81a4dd1f-732d-452e-aa54-50f49e145667 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:07.270228 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81a4dd1f-732d-452e-aa54-50f49e145667 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:07.272519 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81a4dd1f-732d-452e-aa54-50f49e145667 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:07.272832 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81a4dd1f-732d-452e-aa54-50f49e145667 None admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:07.273494 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81a4dd1f-732d-452e-aa54-50f49e145667 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:07.273793 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81a4dd1f-732d-452e-aa54-50f49e145667 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:07.275159 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81a4dd1f-732d-452e-aa54-50f49e145667 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:07.299820 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 313/605] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:43:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 296 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:07.300912 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81a4dd1f-732d-452e-aa54-50f49e145667 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:07.314724 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff05455a-f2a1-4027-91e2-b1dfa4f15318 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:07.353947 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c916c53-b02c-47da-a993-f9645a0a3198 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:07.354649 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c916c53-b02c-47da-a993-f9645a0a3198 None admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:07.354932 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c916c53-b02c-47da-a993-f9645a0a3198 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:07.355208 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c916c53-b02c-47da-a993-f9645a0a3198 None admin] PATH_INFO: `/v3/projects/013c83476b1340fbb051061833c4a9c6/users/7d2eabf9865847ceb6e44d55126078a8/roles/228232e130c5412a80d774978175d28b` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:07.374543 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c916c53-b02c-47da-a993-f9645a0a3198 None admin] RBAC: Authorizing `identity:create_grant(project_id=013c83476b1340fbb051061833c4a9c6, user_id=7d2eabf9865847ceb6e44d55126078a8, role_id=228232e130c5412a80d774978175d28b)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:07.376097 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c916c53-b02c-47da-a993-f9645a0a3198 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:07.394338 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 293/606] 10.222.0.33 () {58 vars in 1439 bytes} [Sun Oct 9 00:43:07 2022] PUT /identity/v3/projects/013c83476b1340fbb051061833c4a9c6/users/7d2eabf9865847ceb6e44d55126078a8/roles/228232e130c5412a80d774978175d28b => generated 0 bytes in 263 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 09 00:43:07.395245 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c916c53-b02c-47da-a993-f9645a0a3198 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:07.447728 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81a4dd1f-732d-452e-aa54-50f49e145667 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:07.476150 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:07.486926 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2348da41-b497-490f-8bc4-9e2fcdfcb705 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:07.487576 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2348da41-b497-490f-8bc4-9e2fcdfcb705 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:07.487886 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2348da41-b497-490f-8bc4-9e2fcdfcb705 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:07.488164 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2348da41-b497-490f-8bc4-9e2fcdfcb705 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:07.500352 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2348da41-b497-490f-8bc4-9e2fcdfcb705 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:43:07.505841 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-2348da41-b497-490f-8bc4-9e2fcdfcb705 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 09 00:43:07.505841 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:07.505841 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:07.505841 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:07.505841 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:07.505841 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:07.505841 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:07.505841 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:07.505841 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:07.505841 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:07.505841 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:07.505841 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:07.505841 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:43:07.505841 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:43:07.505841 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:43:07.505841 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:43:07.506828 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:43:07.506828 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:43:07.506828 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:43:07.506828 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:43:07.506828 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:07.506828 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:07.506828 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:07.506828 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:07.506828 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:07.506828 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:07.506828 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:07.506828 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:07.506828 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:07.506828 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:07.506828 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:07.506828 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:07.508652 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:07.508652 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:07.508652 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:07.508652 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:07.508652 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:43:07.508652 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:43:07.508652 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:43:07.508652 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:43:07.508652 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:43:07.508652 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:43:07.508652 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 09 00:43:07.508652 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:07.508652 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 314/607] 10.222.0.33 () {56 vars in 1102 bytes} [Sun Oct 9 00:43:07 2022] GET /identity/v3/roles/service => generated 85 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:07.513807 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2348da41-b497-490f-8bc4-9e2fcdfcb705 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:07.576577 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:07.628809 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c916c53-b02c-47da-a993-f9645a0a3198 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:07.644090 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2348da41-b497-490f-8bc4-9e2fcdfcb705 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:07.676448 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c249b8b-4f4c-4da8-a25e-7b5167025af8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:07.677266 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c249b8b-4f4c-4da8-a25e-7b5167025af8 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:07.677554 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c249b8b-4f4c-4da8-a25e-7b5167025af8 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:07.678002 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c249b8b-4f4c-4da8-a25e-7b5167025af8 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:07.687296 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c249b8b-4f4c-4da8-a25e-7b5167025af8 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:07.687855 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2f042ef-8aff-415a-a9b4-53e9d2f9be3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:07.688469 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f042ef-8aff-415a-a9b4-53e9d2f9be3a None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:07.688790 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f042ef-8aff-415a-a9b4-53e9d2f9be3a None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:07.689074 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f042ef-8aff-415a-a9b4-53e9d2f9be3a None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:07.689430 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c249b8b-4f4c-4da8-a25e-7b5167025af8 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:07.689732 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2f042ef-8aff-415a-a9b4-53e9d2f9be3a None admin] RBAC: Adding query filter params (name=service) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:07.690097 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2f042ef-8aff-415a-a9b4-53e9d2f9be3a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:07.691618 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2f042ef-8aff-415a-a9b4-53e9d2f9be3a None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:07.701618 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 315/608] 10.222.0.33 () {56 vars in 1103 bytes} [Sun Oct 9 00:43:07 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:43:07.702927 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-2c249b8b-4f4c-4da8-a25e-7b5167025af8 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 09 00:43:07.702927 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:07.702927 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:07.702927 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:07.702927 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:07.702927 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:07.702927 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:07.702927 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:07.702927 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:07.702927 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:07.702927 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:07.702927 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:07.702927 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 09 00:43:07.702927 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 09 00:43:07.702927 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 09 00:43:07.703914 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 09 00:43:07.703914 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:07.703914 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:07.703914 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:07.703914 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:07.703914 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:07.703914 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:07.703914 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:07.703914 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:07.703914 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:07.703914 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:07.703914 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:07.703914 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:07.703914 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:07.703914 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:07.703914 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:07.703914 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:07.705248 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 09 00:43:07.705248 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 09 00:43:07.705248 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 09 00:43:07.705248 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 09 00:43:07.705248 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 09 00:43:07.705248 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 09 00:43:07.705248 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 09 00:43:07.705248 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:07.705248 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 294/609] 10.222.0.33 () {56 vars in 1111 bytes} [Sun Oct 9 00:43:07 2022] GET /identity/v3/projects/service => generated 88 bytes in 309 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:07.706961 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2f042ef-8aff-415a-a9b4-53e9d2f9be3a None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:07.712443 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c249b8b-4f4c-4da8-a25e-7b5167025af8 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:07.874416 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2f042ef-8aff-415a-a9b4-53e9d2f9be3a None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:07.875670 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:07.887042 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c249b8b-4f4c-4da8-a25e-7b5167025af8 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:07.895706 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:07.930992 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-636dac8d-6fd2-43f6-9e6a-f873b2422520 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:07.931646 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-636dac8d-6fd2-43f6-9e6a-f873b2422520 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:07.931933 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-636dac8d-6fd2-43f6-9e6a-f873b2422520 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:07.932206 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-636dac8d-6fd2-43f6-9e6a-f873b2422520 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:07.948226 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6e6b0fa-7eed-46d1-bf3a-1f0cb4b6adc7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:07.948846 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e6b0fa-7eed-46d1-bf3a-1f0cb4b6adc7 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:07.949128 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e6b0fa-7eed-46d1-bf3a-1f0cb4b6adc7 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:07.949400 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e6b0fa-7eed-46d1-bf3a-1f0cb4b6adc7 None admin] PATH_INFO: `/v3/projects` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:07.950094 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6e6b0fa-7eed-46d1-bf3a-1f0cb4b6adc7 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:07.950402 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6e6b0fa-7eed-46d1-bf3a-1f0cb4b6adc7 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:07.951846 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6e6b0fa-7eed-46d1-bf3a-1f0cb4b6adc7 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:07.957263 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-636dac8d-6fd2-43f6-9e6a-f873b2422520 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:07.958738 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-636dac8d-6fd2-43f6-9e6a-f873b2422520 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:07.980139 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-636dac8d-6fd2-43f6-9e6a-f873b2422520 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:07.980139 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:07.980139 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:07.980139 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:07.980139 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:07.980139 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:07.980139 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:07.980139 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:07.980139 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:07.980139 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:07.980139 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:07.980139 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:07.980139 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:43:07.980139 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:43:07.980139 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:43:07.980139 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:43:07.981083 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:07.981083 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:07.981083 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:07.981083 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:07.981083 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:07.981083 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:07.981083 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:07.981083 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:07.981083 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:07.981083 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:07.981083 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:07.981083 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:07.981083 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:07.981083 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:07.981083 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:07.981083 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:07.981083 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:43:07.981956 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:43:07.981956 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:07.981956 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:07.981956 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 316/610] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:43:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 275 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:07.982759 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 295/611] 10.222.0.33 () {56 vars in 1148 bytes} [Sun Oct 9 00:43:07 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 270 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:07.988101 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-636dac8d-6fd2-43f6-9e6a-f873b2422520 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:07.988306 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6e6b0fa-7eed-46d1-bf3a-1f0cb4b6adc7 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:08.175501 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6e6b0fa-7eed-46d1-bf3a-1f0cb4b6adc7 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:08.192662 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-636dac8d-6fd2-43f6-9e6a-f873b2422520 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:08.242410 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-594025e1-30b1-454d-9891-3279b4663080 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:08.243080 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-594025e1-30b1-454d-9891-3279b4663080 None admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:08.243361 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-594025e1-30b1-454d-9891-3279b4663080 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:08.243653 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-594025e1-30b1-454d-9891-3279b4663080 None admin] PATH_INFO: `/v3/projects/013c83476b1340fbb051061833c4a9c6/users/f3f48335668a4e8f934ac11ff66cd308/roles/228232e130c5412a80d774978175d28b` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:08.253327 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61ef9a44-6d9d-4e75-a45d-aecfb797c76f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:08.253947 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ef9a44-6d9d-4e75-a45d-aecfb797c76f None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:08.254213 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ef9a44-6d9d-4e75-a45d-aecfb797c76f None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:08.254482 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ef9a44-6d9d-4e75-a45d-aecfb797c76f None admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:08.255121 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61ef9a44-6d9d-4e75-a45d-aecfb797c76f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:08.255423 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61ef9a44-6d9d-4e75-a45d-aecfb797c76f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:08.256836 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61ef9a44-6d9d-4e75-a45d-aecfb797c76f None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:08.263538 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-594025e1-30b1-454d-9891-3279b4663080 None admin] RBAC: Authorizing `identity:create_grant(project_id=013c83476b1340fbb051061833c4a9c6, user_id=f3f48335668a4e8f934ac11ff66cd308, role_id=228232e130c5412a80d774978175d28b)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:08.265676 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-594025e1-30b1-454d-9891-3279b4663080 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:08.269810 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 317/612] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:43:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 282 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:08.271348 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c5ace0-3684-4b58-bb2c-be485afb926a None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:08.271773 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c5ace0-3684-4b58-bb2c-be485afb926a None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:08.271959 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c5ace0-3684-4b58-bb2c-be485afb926a None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:08.274235 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 318/613] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:43:08 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:43:08.277349 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83c5ace0-3684-4b58-bb2c-be485afb926a None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:08.284815 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 296/614] 10.222.0.33 () {58 vars in 1439 bytes} [Sun Oct 9 00:43:07 2022] PUT /identity/v3/projects/013c83476b1340fbb051061833c4a9c6/users/f3f48335668a4e8f934ac11ff66cd308/roles/228232e130c5412a80d774978175d28b => generated 0 bytes in 297 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 09 00:43:08.286332 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-633a7f95-b0cb-4ac0-bc00-9a85f8909c56 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:08.286798 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-633a7f95-b0cb-4ac0-bc00-9a85f8909c56 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:08.287281 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-633a7f95-b0cb-4ac0-bc00-9a85f8909c56 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:08.294954 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:08.319359 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:08.326342 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-633a7f95-b0cb-4ac0-bc00-9a85f8909c56 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:43:08.328729 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-633a7f95-b0cb-4ac0-bc00-9a85f8909c56 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:43:08.478913 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 297/615] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:43:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 09 00:43:08.481703 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83c5ace0-3684-4b58-bb2c-be485afb926a None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:08.544513 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63e234c2-c036-4be1-8be0-dd4279e3324f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:08.545142 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e234c2-c036-4be1-8be0-dd4279e3324f None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:08.545422 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e234c2-c036-4be1-8be0-dd4279e3324f None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:08.545692 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e234c2-c036-4be1-8be0-dd4279e3324f None admin] PATH_INFO: `/v3/users/swift` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:08.558294 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63e234c2-c036-4be1-8be0-dd4279e3324f None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:08.559778 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63e234c2-c036-4be1-8be0-dd4279e3324f None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:08.573648 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-63e234c2-c036-4be1-8be0-dd4279e3324f None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 09 00:43:08.573648 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:08.573648 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:08.573648 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:08.573648 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:08.573648 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:08.573648 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:08.573648 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:08.573648 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:08.573648 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:08.573648 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:08.573648 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:08.573648 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 09 00:43:08.573648 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 09 00:43:08.573648 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 09 00:43:08.573648 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 09 00:43:08.574640 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:08.574640 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:08.574640 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 09 00:43:08.574640 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 09 00:43:08.574640 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 09 00:43:08.574640 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 09 00:43:08.574640 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:08.574640 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:08.574640 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:08.574640 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:08.574640 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:08.574640 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:08.574640 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:08.574640 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:08.574640 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:08.574640 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:08.574640 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:08.574640 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:08.575663 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:08.575663 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:08.575663 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 09 00:43:08.575663 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 09 00:43:08.575663 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 09 00:43:08.575663 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 09 00:43:08.575663 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 09 00:43:08.575663 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 09 00:43:08.575663 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 09 00:43:08.575663 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:08.575663 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 319/616] 10.222.0.33 () {56 vars in 1096 bytes} [Sun Oct 9 00:43:08 2022] GET /identity/v3/users/swift => generated 83 bytes in 299 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:08.582115 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-633a7f95-b0cb-4ac0-bc00-9a85f8909c56 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:08.706253 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-633a7f95-b0cb-4ac0-bc00-9a85f8909c56 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:08.744909 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:08.750315 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fff82ad4-d767-4cb0-a48e-fbe0e063d60a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:08.750924 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff82ad4-d767-4cb0-a48e-fbe0e063d60a None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:08.751199 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff82ad4-d767-4cb0-a48e-fbe0e063d60a None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:08.751508 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff82ad4-d767-4cb0-a48e-fbe0e063d60a None admin] PATH_INFO: `/v3/users` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:08.752254 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fff82ad4-d767-4cb0-a48e-fbe0e063d60a None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:08.752609 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fff82ad4-d767-4cb0-a48e-fbe0e063d60a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:08.754175 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fff82ad4-d767-4cb0-a48e-fbe0e063d60a None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:08.773456 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 298/617] 10.222.0.33 () {56 vars in 1135 bytes} [Sun Oct 9 00:43:08 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:08.778575 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63e234c2-c036-4be1-8be0-dd4279e3324f None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:08.822643 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fff82ad4-d767-4cb0-a48e-fbe0e063d60a None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:08.923325 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:08.950496 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fff82ad4-d767-4cb0-a48e-fbe0e063d60a None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:08.961984 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63e234c2-c036-4be1-8be0-dd4279e3324f None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:09.001094 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aafaeaf-9870-4233-a871-185787a738c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:09.001957 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aafaeaf-9870-4233-a871-185787a738c2 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:09.002249 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aafaeaf-9870-4233-a871-185787a738c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:09.002530 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aafaeaf-9870-4233-a871-185787a738c2 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:09.021495 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aafaeaf-9870-4233-a871-185787a738c2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:43:09.026713 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d9e69b-168e-4d11-bd95-e152588a02b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:09.027536 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d9e69b-168e-4d11-bd95-e152588a02b9 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:09.027684 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-1aafaeaf-9870-4233-a871-185787a738c2 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 09 00:43:09.027684 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:09.027684 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:09.027684 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:09.027684 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:09.027684 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:09.027684 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:09.027684 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:09.027684 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:09.027684 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:09.027684 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:09.027684 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:09.027684 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:43:09.027684 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:43:09.027684 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:43:09.027684 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:43:09.028751 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:43:09.028751 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:43:09.028751 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:43:09.028751 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:43:09.028751 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:09.028751 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:09.028751 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:09.028751 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:09.028751 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:09.028751 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:09.028751 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:09.028751 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:09.028751 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:09.028751 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:09.028751 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:09.028751 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:09.029769 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:09.029769 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:09.029769 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:09.029769 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:09.029769 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:43:09.029769 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:43:09.029769 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:43:09.029769 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:43:09.029769 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:43:09.029769 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:43:09.029769 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 09 00:43:09.029769 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:09.030968 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d9e69b-168e-4d11-bd95-e152588a02b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:09.031283 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 299/618] 10.222.0.33 () {56 vars in 1102 bytes} [Sun Oct 9 00:43:08 2022] GET /identity/v3/roles/service => generated 85 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 09 00:43:09.031549 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d9e69b-168e-4d11-bd95-e152588a02b9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:09.040370 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aafaeaf-9870-4233-a871-185787a738c2 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:09.060003 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35d9e69b-168e-4d11-bd95-e152588a02b9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:09.061564 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35d9e69b-168e-4d11-bd95-e152588a02b9 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:09.078923 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-35d9e69b-168e-4d11-bd95-e152588a02b9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:09.078923 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:09.078923 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:09.078923 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:09.078923 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:09.078923 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:09.078923 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:09.078923 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:09.078923 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:09.078923 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:09.078923 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:09.078923 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:09.078923 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:43:09.078923 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:43:09.078923 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:43:09.078923 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:43:09.080286 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:09.080286 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:09.080286 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:09.080286 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:09.080286 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:09.080286 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:09.080286 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:09.080286 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:09.080286 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:09.080286 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:09.080286 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:09.080286 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:09.080286 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:09.080286 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:09.080286 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:09.080286 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:09.080286 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:43:09.081406 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:43:09.081406 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:09.081406 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:09.081406 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 320/619] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:43:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 302 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 09 00:43:09.082185 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78c0db5a-d876-4437-ada6-e1b85c6405ff None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:09.082185 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78c0db5a-d876-4437-ada6-e1b85c6405ff None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:09.082185 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78c0db5a-d876-4437-ada6-e1b85c6405ff None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:09.092045 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 321/620] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:43:09 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:43:09.092045 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78c0db5a-d876-4437-ada6-e1b85c6405ff None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:09.116392 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:09.172494 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aafaeaf-9870-4233-a871-185787a738c2 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:09.209354 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-435b7819-d192-40b0-a09c-f3e86ad76945 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:09.209978 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78c0db5a-d876-4437-ada6-e1b85c6405ff None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:09.210110 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-435b7819-d192-40b0-a09c-f3e86ad76945 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:09.210256 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-435b7819-d192-40b0-a09c-f3e86ad76945 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:09.210581 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-435b7819-d192-40b0-a09c-f3e86ad76945 None admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:09.211236 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-435b7819-d192-40b0-a09c-f3e86ad76945 None admin] RBAC: Adding query filter params (name=service) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:09.211607 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-435b7819-d192-40b0-a09c-f3e86ad76945 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:09.213862 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-435b7819-d192-40b0-a09c-f3e86ad76945 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:09.226642 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 300/621] 10.222.0.33 () {56 vars in 1103 bytes} [Sun Oct 9 00:43:09 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:09.226642 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f663d7-1d59-41fa-9ca3-fd42446f8327 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:09.226642 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f663d7-1d59-41fa-9ca3-fd42446f8327 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:09.226642 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f663d7-1d59-41fa-9ca3-fd42446f8327 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:09.243662 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-a3f663d7-1d59-41fa-9ca3-fd42446f8327 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:43:09.245009 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-a3f663d7-1d59-41fa-9ca3-fd42446f8327 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:43:09.253963 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-354cd94c-ba56-4084-bd8b-d5c368abe2b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:09.254759 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-354cd94c-ba56-4084-bd8b-d5c368abe2b3 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:09.255279 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-354cd94c-ba56-4084-bd8b-d5c368abe2b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:09.255724 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-354cd94c-ba56-4084-bd8b-d5c368abe2b3 None admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:09.256627 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-354cd94c-ba56-4084-bd8b-d5c368abe2b3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:09.257081 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-354cd94c-ba56-4084-bd8b-d5c368abe2b3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:09.259206 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-354cd94c-ba56-4084-bd8b-d5c368abe2b3 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:09.262022 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:09.279345 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 322/622] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:43:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:09.280288 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-354cd94c-ba56-4084-bd8b-d5c368abe2b3 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:09.320473 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 301/623] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:43:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 09 00:43:09.321568 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3f663d7-1d59-41fa-9ca3-fd42446f8327 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:09.382298 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:09.425487 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-354cd94c-ba56-4084-bd8b-d5c368abe2b3 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:09.471868 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c584e558-46e9-4e90-a7a3-037ecfc0449f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:09.472503 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c584e558-46e9-4e90-a7a3-037ecfc0449f None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:09.472781 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c584e558-46e9-4e90-a7a3-037ecfc0449f None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:09.473052 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c584e558-46e9-4e90-a7a3-037ecfc0449f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:09.495167 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c584e558-46e9-4e90-a7a3-037ecfc0449f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:09.497454 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3f663d7-1d59-41fa-9ca3-fd42446f8327 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:09.500828 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c584e558-46e9-4e90-a7a3-037ecfc0449f None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:09.526650 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-c584e558-46e9-4e90-a7a3-037ecfc0449f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:09.526650 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:09.526650 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:09.526650 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:09.526650 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:09.526650 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:09.526650 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:09.526650 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:09.526650 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:09.526650 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:09.526650 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:09.526650 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:09.526650 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:43:09.526650 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:43:09.526650 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:43:09.526650 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:43:09.527632 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:09.527632 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:09.527632 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:09.527632 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:09.527632 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:09.527632 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:09.527632 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:09.527632 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:09.527632 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:09.527632 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:09.527632 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:09.527632 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:09.527632 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:09.527632 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:09.527632 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:09.527632 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:09.527632 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:43:09.528517 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:43:09.528517 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:09.528517 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:09.531909 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 323/624] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:43:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 252 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:09.539070 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c584e558-46e9-4e90-a7a3-037ecfc0449f None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:09.569166 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaacb6fd-0ba8-4859-b49a-4beabfe71a0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:09.570193 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaacb6fd-0ba8-4859-b49a-4beabfe71a0b None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:09.570673 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaacb6fd-0ba8-4859-b49a-4beabfe71a0b None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:09.571168 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaacb6fd-0ba8-4859-b49a-4beabfe71a0b None admin] PATH_INFO: `/v3/projects/service` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:09.584746 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaacb6fd-0ba8-4859-b49a-4beabfe71a0b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:09.587680 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaacb6fd-0ba8-4859-b49a-4beabfe71a0b None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:09.609451 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-aaacb6fd-0ba8-4859-b49a-4beabfe71a0b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 09 00:43:09.609451 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:09.609451 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:09.609451 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:09.609451 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:09.609451 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:09.609451 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:09.609451 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:09.609451 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:09.609451 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:09.609451 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:09.609451 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:09.609451 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 09 00:43:09.609451 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 09 00:43:09.609451 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 09 00:43:09.610346 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 09 00:43:09.610346 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:09.610346 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:09.610346 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:09.610346 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:09.610346 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:09.610346 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:09.610346 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:09.610346 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:09.610346 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:09.610346 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:09.610346 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:09.610346 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:09.610346 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:09.610346 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:09.610346 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:09.610346 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:09.611206 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 09 00:43:09.611206 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 09 00:43:09.611206 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 09 00:43:09.611206 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 09 00:43:09.611206 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 09 00:43:09.611206 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 09 00:43:09.611206 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 09 00:43:09.611206 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:09.611206 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 302/625] 10.222.0.33 () {56 vars in 1111 bytes} [Sun Oct 9 00:43:09 2022] GET /identity/v3/projects/service => generated 88 bytes in 290 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:09.617220 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaacb6fd-0ba8-4859-b49a-4beabfe71a0b None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:09.629150 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:09.694153 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c584e558-46e9-4e90-a7a3-037ecfc0449f None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:09.729743 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd5b74d8-ee7e-4902-b327-cfa28f5d1ec9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:09.730346 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd5b74d8-ee7e-4902-b327-cfa28f5d1ec9 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:09.730628 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd5b74d8-ee7e-4902-b327-cfa28f5d1ec9 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:09.730899 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd5b74d8-ee7e-4902-b327-cfa28f5d1ec9 None admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:09.731618 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd5b74d8-ee7e-4902-b327-cfa28f5d1ec9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:09.731934 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd5b74d8-ee7e-4902-b327-cfa28f5d1ec9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:09.733628 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd5b74d8-ee7e-4902-b327-cfa28f5d1ec9 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:09.746598 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 324/626] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:43:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:09.747249 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaacb6fd-0ba8-4859-b49a-4beabfe71a0b None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:09.747553 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd5b74d8-ee7e-4902-b327-cfa28f5d1ec9 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:09.749451 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:09.795377 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93e035ac-4634-4e6f-ae9f-9d1516081f2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:09.796560 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93e035ac-4634-4e6f-ae9f-9d1516081f2e None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:09.796997 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93e035ac-4634-4e6f-ae9f-9d1516081f2e None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:09.797419 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93e035ac-4634-4e6f-ae9f-9d1516081f2e None admin] PATH_INFO: `/v3/roles/service` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:09.812107 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93e035ac-4634-4e6f-ae9f-9d1516081f2e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:43:09.818822 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-93e035ac-4634-4e6f-ae9f-9d1516081f2e None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 09 00:43:09.818822 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:09.818822 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:09.818822 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:09.818822 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:09.818822 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:09.818822 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:09.818822 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:09.818822 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:09.818822 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:09.818822 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:09.818822 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:09.818822 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:43:09.818822 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:43:09.818822 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:43:09.818822 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:43:09.820093 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:43:09.820093 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:43:09.820093 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:43:09.820093 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:43:09.820093 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:09.820093 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:09.820093 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:09.820093 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:09.820093 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:09.820093 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:09.820093 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:09.820093 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:09.820093 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:09.820093 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:09.820093 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:09.820093 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:09.821155 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:09.821155 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:09.821155 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:09.821155 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:09.821155 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:43:09.821155 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:43:09.821155 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:43:09.821155 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:43:09.821155 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:43:09.821155 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:43:09.821155 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 09 00:43:09.821155 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:09.823575 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 303/627] 10.222.0.33 () {56 vars in 1102 bytes} [Sun Oct 9 00:43:09 2022] GET /identity/v3/roles/service => generated 85 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 09 00:43:09.824713 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93e035ac-4634-4e6f-ae9f-9d1516081f2e None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:09.866012 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd5b74d8-ee7e-4902-b327-cfa28f5d1ec9 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:09.875811 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:09.905170 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-637918cc-cc8b-4f38-8831-6a678b059be7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:09.905939 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-637918cc-cc8b-4f38-8831-6a678b059be7 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:09.906248 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-637918cc-cc8b-4f38-8831-6a678b059be7 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:09.906570 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-637918cc-cc8b-4f38-8831-6a678b059be7 None admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:09.907332 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-637918cc-cc8b-4f38-8831-6a678b059be7 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:09.907740 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-637918cc-cc8b-4f38-8831-6a678b059be7 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:09.909244 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-637918cc-cc8b-4f38-8831-6a678b059be7 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:09.927077 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 325/628] 10.222.0.33 () {56 vars in 1148 bytes} [Sun Oct 9 00:43:09 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:09.927077 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-637918cc-cc8b-4f38-8831-6a678b059be7 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:09.953841 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93e035ac-4634-4e6f-ae9f-9d1516081f2e None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:09.988599 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee7bbbe2-5aac-4e2c-a8a3-7c43fa9025f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:09.989334 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee7bbbe2-5aac-4e2c-a8a3-7c43fa9025f8 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:09.989608 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee7bbbe2-5aac-4e2c-a8a3-7c43fa9025f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:09.989988 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee7bbbe2-5aac-4e2c-a8a3-7c43fa9025f8 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:09.996862 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee7bbbe2-5aac-4e2c-a8a3-7c43fa9025f8 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:09.998277 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee7bbbe2-5aac-4e2c-a8a3-7c43fa9025f8 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:10.009489 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-ee7bbbe2-5aac-4e2c-a8a3-7c43fa9025f8 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 09 00:43:10.009489 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:10.009489 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:10.009489 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:10.009489 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:10.009489 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:10.009489 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:10.009489 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:10.009489 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:10.009489 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:10.009489 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:10.009489 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:10.009489 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 09 00:43:10.009489 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 09 00:43:10.009489 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 09 00:43:10.009489 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 09 00:43:10.010656 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:10.010656 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:10.010656 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 09 00:43:10.010656 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 09 00:43:10.010656 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 09 00:43:10.010656 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 09 00:43:10.010656 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:10.010656 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:10.010656 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:10.010656 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:10.010656 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:10.010656 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:10.010656 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:10.010656 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:10.010656 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:10.010656 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:10.010656 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:10.010656 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:10.011849 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:10.011849 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:10.011849 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 09 00:43:10.011849 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 09 00:43:10.011849 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 09 00:43:10.011849 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 09 00:43:10.011849 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 09 00:43:10.011849 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 09 00:43:10.011849 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 09 00:43:10.011849 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:10.015474 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 304/629] 10.222.0.33 () {56 vars in 1093 bytes} [Sun Oct 9 00:43:09 2022] GET /identity/v3/users/nova => generated 82 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:10.016301 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee7bbbe2-5aac-4e2c-a8a3-7c43fa9025f8 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:10.037100 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-637918cc-cc8b-4f38-8831-6a678b059be7 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:10.043867 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:10.072234 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ae06095-a71e-4eb5-b369-02957bc67a59 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:10.072768 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ae06095-a71e-4eb5-b369-02957bc67a59 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:10.073477 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ae06095-a71e-4eb5-b369-02957bc67a59 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:10.073757 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ae06095-a71e-4eb5-b369-02957bc67a59 None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:10.074416 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ae06095-a71e-4eb5-b369-02957bc67a59 None admin] RBAC: Adding query filter params (name=service) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:10.074719 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ae06095-a71e-4eb5-b369-02957bc67a59 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:10.076118 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ae06095-a71e-4eb5-b369-02957bc67a59 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:10.084612 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 326/630] 10.222.0.33 () {56 vars in 1103 bytes} [Sun Oct 9 00:43:09 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:10.086584 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0de7024-17e4-45c1-a4cc-0c7cb2c112da None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:10.086873 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0de7024-17e4-45c1-a4cc-0c7cb2c112da None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:10.087150 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0de7024-17e4-45c1-a4cc-0c7cb2c112da None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:10.092741 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 327/631] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:43:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:43:10.092741 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0de7024-17e4-45c1-a4cc-0c7cb2c112da None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:10.122318 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:10.138823 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee7bbbe2-5aac-4e2c-a8a3-7c43fa9025f8 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:10.176190 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-526eab2b-77d5-450c-aeb6-997423f15bfe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:10.177188 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-526eab2b-77d5-450c-aeb6-997423f15bfe None admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:10.177768 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-526eab2b-77d5-450c-aeb6-997423f15bfe None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:10.178041 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-526eab2b-77d5-450c-aeb6-997423f15bfe None admin] PATH_INFO: `/v3/projects/013c83476b1340fbb051061833c4a9c6/users/70e367e802894b08afd7d756d8d9ecff/roles/228232e130c5412a80d774978175d28b` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:10.193685 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-526eab2b-77d5-450c-aeb6-997423f15bfe None admin] RBAC: Authorizing `identity:create_grant(project_id=013c83476b1340fbb051061833c4a9c6, user_id=70e367e802894b08afd7d756d8d9ecff, role_id=228232e130c5412a80d774978175d28b)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:10.195665 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-526eab2b-77d5-450c-aeb6-997423f15bfe None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:10.196196 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0de7024-17e4-45c1-a4cc-0c7cb2c112da None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:10.219622 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 305/632] 10.222.0.33 () {58 vars in 1439 bytes} [Sun Oct 9 00:43:10 2022] PUT /identity/v3/projects/013c83476b1340fbb051061833c4a9c6/users/70e367e802894b08afd7d756d8d9ecff/roles/228232e130c5412a80d774978175d28b => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:43:10.219622 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-526eab2b-77d5-450c-aeb6-997423f15bfe None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:10.250107 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d45a252a-62fc-4c85-8ba8-bb8b4ccb3eb3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:10.250883 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d45a252a-62fc-4c85-8ba8-bb8b4ccb3eb3 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:10.251243 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d45a252a-62fc-4c85-8ba8-bb8b4ccb3eb3 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:10.251628 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d45a252a-62fc-4c85-8ba8-bb8b4ccb3eb3 None admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:10.252454 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d45a252a-62fc-4c85-8ba8-bb8b4ccb3eb3 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:10.252824 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d45a252a-62fc-4c85-8ba8-bb8b4ccb3eb3 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:10.254781 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d45a252a-62fc-4c85-8ba8-bb8b4ccb3eb3 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:10.280019 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 328/633] 10.222.0.33 () {56 vars in 1133 bytes} [Sun Oct 9 00:43:10 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:10.280019 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95435a8e-df89-486e-ac01-c8713505b680 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:10.280019 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95435a8e-df89-486e-ac01-c8713505b680 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:10.280019 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95435a8e-df89-486e-ac01-c8713505b680 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:10.303174 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-95435a8e-df89-486e-ac01-c8713505b680 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:43:10.304738 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-95435a8e-df89-486e-ac01-c8713505b680 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:43:10.313006 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:10.354653 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-526eab2b-77d5-450c-aeb6-997423f15bfe None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:10.395778 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 329/634] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:43:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 09 00:43:10.395778 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95435a8e-df89-486e-ac01-c8713505b680 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:10.416241 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5d72467-39df-4174-83c0-71f4ff358c5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:10.417088 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d72467-39df-4174-83c0-71f4ff358c5a None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:10.417208 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d72467-39df-4174-83c0-71f4ff358c5a None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:10.418812 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d72467-39df-4174-83c0-71f4ff358c5a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:10.429005 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:10.431082 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5d72467-39df-4174-83c0-71f4ff358c5a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:10.432642 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5d72467-39df-4174-83c0-71f4ff358c5a None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:10.454631 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-d5d72467-39df-4174-83c0-71f4ff358c5a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:10.454631 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:10.454631 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:10.454631 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:10.454631 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:10.454631 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:10.454631 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:10.454631 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:10.454631 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:10.454631 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:10.454631 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:10.454631 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:10.454631 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:43:10.454631 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:43:10.454631 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:43:10.454631 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:43:10.455842 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:10.455842 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:10.455842 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:10.455842 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:10.455842 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:10.455842 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:10.455842 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:10.455842 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:10.455842 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:10.455842 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:10.455842 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:10.455842 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:10.455842 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:10.455842 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:10.455842 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:10.455842 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:10.455842 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:43:10.456911 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:43:10.456911 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:10.456911 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:10.456911 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 306/635] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:43:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 246 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:10.463789 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5d72467-39df-4174-83c0-71f4ff358c5a None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:10.526516 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95435a8e-df89-486e-ac01-c8713505b680 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:10.564714 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-762cee20-e0b6-4428-a62d-4ad02bea2dad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:10.565629 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-762cee20-e0b6-4428-a62d-4ad02bea2dad None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:10.566059 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-762cee20-e0b6-4428-a62d-4ad02bea2dad None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:10.566551 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-762cee20-e0b6-4428-a62d-4ad02bea2dad None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:10.587647 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5d72467-39df-4174-83c0-71f4ff358c5a None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:10.617386 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:10.620165 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f181c7c1-2f1b-4b7b-9558-34a03bf74aed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:10.620879 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f181c7c1-2f1b-4b7b-9558-34a03bf74aed None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:10.621359 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f181c7c1-2f1b-4b7b-9558-34a03bf74aed None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:10.622075 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f181c7c1-2f1b-4b7b-9558-34a03bf74aed None admin] PATH_INFO: `/v3/domains` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:10.622795 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f181c7c1-2f1b-4b7b-9558-34a03bf74aed None admin] RBAC: Adding query filter params (name=Default) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:10.623324 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f181c7c1-2f1b-4b7b-9558-34a03bf74aed None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:10.624852 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f181c7c1-2f1b-4b7b-9558-34a03bf74aed None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:10.638509 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 307/636] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:43:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:10.644252 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f181c7c1-2f1b-4b7b-9558-34a03bf74aed None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:10.700367 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-762cee20-e0b6-4428-a62d-4ad02bea2dad None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:10.702102 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-762cee20-e0b6-4428-a62d-4ad02bea2dad None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:10.736593 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:10.756313 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f181c7c1-2f1b-4b7b-9558-34a03bf74aed None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:10.798630 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6db62f6e-5fe7-4fc5-8142-cfaff5824ba3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:10.799270 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6db62f6e-5fe7-4fc5-8142-cfaff5824ba3 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:10.799550 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6db62f6e-5fe7-4fc5-8142-cfaff5824ba3 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:10.800018 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6db62f6e-5fe7-4fc5-8142-cfaff5824ba3 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:10.810807 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6db62f6e-5fe7-4fc5-8142-cfaff5824ba3 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:10.813573 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6db62f6e-5fe7-4fc5-8142-cfaff5824ba3 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:10.830328 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-6db62f6e-5fe7-4fc5-8142-cfaff5824ba3 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Oct 09 00:43:10.830328 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:10.830328 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:10.830328 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:10.830328 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:10.830328 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:10.830328 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:10.830328 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:10.830328 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:10.830328 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:10.830328 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:10.830328 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:10.830328 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 09 00:43:10.830328 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 09 00:43:10.830328 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 09 00:43:10.830328 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 09 00:43:10.831340 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:10.831340 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:10.831340 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 09 00:43:10.831340 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 09 00:43:10.831340 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 09 00:43:10.831340 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 09 00:43:10.831340 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:10.831340 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:10.831340 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:10.831340 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:10.831340 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:10.831340 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:10.831340 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:10.831340 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:10.831340 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:10.831340 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:10.831340 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:10.831340 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:10.832284 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:10.832284 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:10.832284 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 09 00:43:10.832284 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 09 00:43:10.832284 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 09 00:43:10.832284 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 09 00:43:10.832284 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 09 00:43:10.832284 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 09 00:43:10.832284 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Oct 09 00:43:10.832284 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:10.836110 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 308/637] 10.222.0.33 () {56 vars in 1099 bytes} [Sun Oct 9 00:43:10 2022] GET /identity/v3/users/glance => generated 84 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:10.837216 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6db62f6e-5fe7-4fc5-8142-cfaff5824ba3 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:10.865040 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 330/638] 10.222.0.33 () {58 vars in 1282 bytes} [Sun Oct 9 00:43:10 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 473 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 09 00:43:10.866225 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-762cee20-e0b6-4428-a62d-4ad02bea2dad None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:10.893104 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:10.948013 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6db62f6e-5fe7-4fc5-8142-cfaff5824ba3 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:10.982146 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-762cee20-e0b6-4428-a62d-4ad02bea2dad None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:10.983837 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dee8256-ea10-4ca7-be4c-14f57498d0df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:10.984473 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dee8256-ea10-4ca7-be4c-14f57498d0df None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:10.984808 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dee8256-ea10-4ca7-be4c-14f57498d0df None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:10.985088 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dee8256-ea10-4ca7-be4c-14f57498d0df None admin] PATH_INFO: `/v3/roles/service` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:10.995319 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dee8256-ea10-4ca7-be4c-14f57498d0df None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:43:11.000030 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:11.001485 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-4dee8256-ea10-4ca7-be4c-14f57498d0df None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 09 00:43:11.001485 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:11.001485 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:11.001485 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:11.001485 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:11.001485 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:11.001485 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:11.001485 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:11.001485 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:11.001485 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:11.001485 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:11.001485 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:11.001485 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:43:11.001485 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:43:11.001485 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:43:11.001485 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:43:11.002415 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:43:11.002415 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:43:11.002415 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:43:11.002415 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:43:11.002415 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:11.002415 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:11.002415 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:11.002415 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:11.002415 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:11.002415 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:11.002415 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:11.002415 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:11.002415 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:11.002415 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:11.002415 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:11.002415 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:11.009437 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:11.009437 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:11.009437 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:11.009437 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:11.009437 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:43:11.009437 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:43:11.009437 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:43:11.009437 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:43:11.009437 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:43:11.009437 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:43:11.009437 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 09 00:43:11.009437 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:11.009437 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 309/639] 10.222.0.33 () {56 vars in 1102 bytes} [Sun Oct 9 00:43:10 2022] GET /identity/v3/roles/service => generated 85 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:11.009437 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3623f506-b63c-48a9-9d1d-14147f1649af None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:11.010323 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3623f506-b63c-48a9-9d1d-14147f1649af None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:11.010323 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3623f506-b63c-48a9-9d1d-14147f1649af None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:11.010323 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 310/640] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:43:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:43:11.010323 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3623f506-b63c-48a9-9d1d-14147f1649af None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:11.025110 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a999d837-e72d-47b9-b59d-5ba9fc3028f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:11.025711 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a999d837-e72d-47b9-b59d-5ba9fc3028f4 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:11.025991 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a999d837-e72d-47b9-b59d-5ba9fc3028f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:11.026252 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a999d837-e72d-47b9-b59d-5ba9fc3028f4 None admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:11.026970 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a999d837-e72d-47b9-b59d-5ba9fc3028f4 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:11.027271 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a999d837-e72d-47b9-b59d-5ba9fc3028f4 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:11.028732 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a999d837-e72d-47b9-b59d-5ba9fc3028f4 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:11.051497 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 331/641] 10.222.0.33 () {56 vars in 1137 bytes} [Sun Oct 9 00:43:10 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:11.051497 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a999d837-e72d-47b9-b59d-5ba9fc3028f4 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:11.130995 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3623f506-b63c-48a9-9d1d-14147f1649af None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:11.134739 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:11.168316 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b215d7f-83c9-44f7-b15d-7ba48f5513a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:11.169089 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b215d7f-83c9-44f7-b15d-7ba48f5513a3 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:11.169373 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b215d7f-83c9-44f7-b15d-7ba48f5513a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:11.169647 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b215d7f-83c9-44f7-b15d-7ba48f5513a3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:11.170661 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a999d837-e72d-47b9-b59d-5ba9fc3028f4 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:11.180173 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b215d7f-83c9-44f7-b15d-7ba48f5513a3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:11.181571 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b215d7f-83c9-44f7-b15d-7ba48f5513a3 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:11.197517 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-4b215d7f-83c9-44f7-b15d-7ba48f5513a3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:11.197517 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:11.197517 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:11.197517 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:11.197517 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:11.197517 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:11.197517 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:11.197517 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:11.197517 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:11.197517 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:11.197517 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:11.197517 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:11.197517 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:43:11.197517 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:43:11.197517 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:43:11.197517 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:43:11.198541 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:11.198541 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:11.198541 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:11.198541 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:11.198541 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:11.198541 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:11.198541 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:11.198541 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:11.198541 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:11.198541 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:11.198541 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:11.198541 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:11.198541 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:11.198541 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:11.198541 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:11.198541 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:11.198541 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:43:11.199426 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:43:11.199426 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:11.199426 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:11.199426 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 311/642] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:43:11 2022] GET /identity/v3/domains/Default => generated 87 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:11.204717 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4e3f13a-64df-459c-9493-bdcb1c2e58dd None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:11.204717 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4e3f13a-64df-459c-9493-bdcb1c2e58dd None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:11.204717 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4e3f13a-64df-459c-9493-bdcb1c2e58dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:11.214709 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1facbadb-997e-4ba0-a20d-9d7e07b888cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:11.215312 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1facbadb-997e-4ba0-a20d-9d7e07b888cc None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:11.215617 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1facbadb-997e-4ba0-a20d-9d7e07b888cc None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:11.215898 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1facbadb-997e-4ba0-a20d-9d7e07b888cc None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:11.216554 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1facbadb-997e-4ba0-a20d-9d7e07b888cc None admin] RBAC: Adding query filter params (name=service) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:11.216851 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1facbadb-997e-4ba0-a20d-9d7e07b888cc None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:11.218236 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1facbadb-997e-4ba0-a20d-9d7e07b888cc None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:11.221632 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-d4e3f13a-64df-459c-9493-bdcb1c2e58dd None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:43:11.223031 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-d4e3f13a-64df-459c-9493-bdcb1c2e58dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:43:11.230842 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 332/643] 10.222.0.33 () {56 vars in 1103 bytes} [Sun Oct 9 00:43:11 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:11.230842 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1facbadb-997e-4ba0-a20d-9d7e07b888cc None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:11.267890 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:11.302303 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 312/644] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:43:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 09 00:43:11.303054 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4e3f13a-64df-459c-9493-bdcb1c2e58dd None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:11.346982 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1facbadb-997e-4ba0-a20d-9d7e07b888cc None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:11.387782 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd74f44e-a423-4d85-a168-1a1dbf574fd4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:11.388434 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd74f44e-a423-4d85-a168-1a1dbf574fd4 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:11.388760 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd74f44e-a423-4d85-a168-1a1dbf574fd4 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:11.389041 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd74f44e-a423-4d85-a168-1a1dbf574fd4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:11.403505 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:11.425992 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4e3f13a-64df-459c-9493-bdcb1c2e58dd None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:11.460902 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd74f44e-a423-4d85-a168-1a1dbf574fd4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:11.462353 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd74f44e-a423-4d85-a168-1a1dbf574fd4 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:11.472813 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-290c5ecc-13e2-4607-bcff-86c1b79feb22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:11.474123 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-290c5ecc-13e2-4607-bcff-86c1b79feb22 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:11.474899 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-290c5ecc-13e2-4607-bcff-86c1b79feb22 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:11.475378 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-290c5ecc-13e2-4607-bcff-86c1b79feb22 None admin] PATH_INFO: `/v3/domains` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:11.476560 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-290c5ecc-13e2-4607-bcff-86c1b79feb22 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:11.477550 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-290c5ecc-13e2-4607-bcff-86c1b79feb22 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:11.479700 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-290c5ecc-13e2-4607-bcff-86c1b79feb22 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:11.497099 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 313/645] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:43:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:11.498218 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-290c5ecc-13e2-4607-bcff-86c1b79feb22 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:11.508035 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:11.574746 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 333/646] 10.222.0.33 () {58 vars in 1282 bytes} [Sun Oct 9 00:43:11 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 348 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 09 00:43:11.575585 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd74f44e-a423-4d85-a168-1a1dbf574fd4 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:11.621169 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-290c5ecc-13e2-4607-bcff-86c1b79feb22 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:11.655816 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85e91c53-4247-4518-a383-244dfab41007 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:11.656442 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e91c53-4247-4518-a383-244dfab41007 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:11.656720 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e91c53-4247-4518-a383-244dfab41007 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:11.656994 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e91c53-4247-4518-a383-244dfab41007 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:11.668590 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85e91c53-4247-4518-a383-244dfab41007 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:11.670053 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85e91c53-4247-4518-a383-244dfab41007 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:11.676556 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:11.685689 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd74f44e-a423-4d85-a168-1a1dbf574fd4 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:11.685942 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-85e91c53-4247-4518-a383-244dfab41007 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:11.685942 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:11.685942 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:11.685942 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:11.685942 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:11.685942 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:11.685942 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:11.685942 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:11.685942 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:11.685942 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:11.685942 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:11.685942 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:11.685942 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:43:11.685942 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:43:11.685942 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:43:11.685942 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:43:11.686907 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:11.686907 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:11.686907 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:11.686907 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:11.686907 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:11.686907 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:11.686907 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:11.686907 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:11.686907 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:11.686907 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:11.686907 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:11.686907 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:11.686907 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:11.686907 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:11.686907 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:11.686907 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:11.686907 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:43:11.688454 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:43:11.688454 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:11.688454 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:11.688454 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 314/647] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:43:11 2022] GET /identity/v3/domains/Default => generated 87 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:11.688454 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85e91c53-4247-4518-a383-244dfab41007 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:11.726694 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcfd8cef-4a3e-42fa-9a23-a838b5431275 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:11.727323 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcfd8cef-4a3e-42fa-9a23-a838b5431275 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:11.727616 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcfd8cef-4a3e-42fa-9a23-a838b5431275 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:11.727913 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcfd8cef-4a3e-42fa-9a23-a838b5431275 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:11.734493 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcfd8cef-4a3e-42fa-9a23-a838b5431275 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:11.736034 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcfd8cef-4a3e-42fa-9a23-a838b5431275 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:11.737834 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:11.748621 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-bcfd8cef-4a3e-42fa-9a23-a838b5431275 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 09 00:43:11.748621 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:11.748621 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:11.748621 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:11.748621 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:11.748621 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:11.748621 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:11.748621 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:11.748621 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:11.748621 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:11.748621 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:11.748621 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:11.748621 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 09 00:43:11.748621 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 09 00:43:11.748621 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 09 00:43:11.749666 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 09 00:43:11.749666 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:11.749666 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:11.749666 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:11.749666 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:11.749666 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:11.749666 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:11.749666 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:11.749666 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:11.749666 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:11.749666 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:11.749666 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:11.749666 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:11.749666 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:11.749666 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:11.749666 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:11.749666 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:11.750824 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 09 00:43:11.750824 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 09 00:43:11.750824 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 09 00:43:11.750824 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 09 00:43:11.750824 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 09 00:43:11.750824 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 09 00:43:11.750824 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 09 00:43:11.750824 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:11.757242 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 334/648] 10.222.0.33 () {56 vars in 1111 bytes} [Sun Oct 9 00:43:11 2022] GET /identity/v3/projects/service => generated 88 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:11.757242 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcfd8cef-4a3e-42fa-9a23-a838b5431275 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:11.806500 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85e91c53-4247-4518-a383-244dfab41007 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:11.841875 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8fc1ab1-f654-40d1-9ad6-7821cca6f678 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:11.842716 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8fc1ab1-f654-40d1-9ad6-7821cca6f678 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:11.842979 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8fc1ab1-f654-40d1-9ad6-7821cca6f678 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:11.843312 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8fc1ab1-f654-40d1-9ad6-7821cca6f678 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:11.853995 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8fc1ab1-f654-40d1-9ad6-7821cca6f678 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:43:11.858860 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcfd8cef-4a3e-42fa-9a23-a838b5431275 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:11.859317 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-a8fc1ab1-f654-40d1-9ad6-7821cca6f678 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 09 00:43:11.859317 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:11.859317 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:11.859317 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:11.859317 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:11.859317 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:11.859317 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:11.859317 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:11.859317 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:11.859317 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:11.859317 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:11.859317 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:11.859317 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:43:11.859317 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:43:11.859317 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:43:11.859317 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:43:11.860519 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:43:11.860519 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:43:11.860519 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:43:11.860519 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:43:11.860519 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:11.860519 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:11.860519 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:11.860519 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:11.860519 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:11.860519 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:11.860519 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:11.860519 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:11.860519 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:11.860519 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:11.860519 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:11.860519 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:11.861427 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:11.861427 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:11.861427 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:11.861427 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:11.861427 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:43:11.861427 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:43:11.861427 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:43:11.861427 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:43:11.861427 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:43:11.861427 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:43:11.861427 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 09 00:43:11.861427 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:11.861427 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 315/649] 10.222.0.33 () {56 vars in 1102 bytes} [Sun Oct 9 00:43:11 2022] GET /identity/v3/roles/service => generated 85 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:11.862143 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8fc1ab1-f654-40d1-9ad6-7821cca6f678 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:11.898848 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3862fb7d-35ff-4c99-af60-5ce8200bf968 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:11.899560 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3862fb7d-35ff-4c99-af60-5ce8200bf968 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:11.899976 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3862fb7d-35ff-4c99-af60-5ce8200bf968 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:11.900309 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3862fb7d-35ff-4c99-af60-5ce8200bf968 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:11.911465 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3862fb7d-35ff-4c99-af60-5ce8200bf968 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:11.913262 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3862fb7d-35ff-4c99-af60-5ce8200bf968 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:11.925119 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:11.928558 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-3862fb7d-35ff-4c99-af60-5ce8200bf968 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:11.928558 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:11.928558 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:11.928558 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:11.928558 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:11.928558 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:11.928558 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:11.928558 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:11.928558 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:11.928558 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:11.928558 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:11.928558 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:11.928558 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:43:11.928558 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:43:11.928558 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:43:11.928558 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:43:11.930151 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:11.930151 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:11.930151 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:11.930151 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:11.930151 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:11.930151 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:11.930151 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:11.930151 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:11.930151 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:11.930151 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:11.930151 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:11.930151 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:11.930151 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:11.930151 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:11.930151 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:11.930151 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:11.930151 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:43:11.931072 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:43:11.931072 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:11.931072 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:11.932278 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 335/650] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:43:11 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:11.933126 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3862fb7d-35ff-4c99-af60-5ce8200bf968 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:11.979894 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8fc1ab1-f654-40d1-9ad6-7821cca6f678 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:11.985370 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:12.017071 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d421d5ee-7b3e-4ca3-8284-ab95ab39bb07 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:12.017669 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d421d5ee-7b3e-4ca3-8284-ab95ab39bb07 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:12.017940 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d421d5ee-7b3e-4ca3-8284-ab95ab39bb07 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:12.018209 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d421d5ee-7b3e-4ca3-8284-ab95ab39bb07 None admin] PATH_INFO: `/v3/domains` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:12.018838 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d421d5ee-7b3e-4ca3-8284-ab95ab39bb07 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:12.019135 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d421d5ee-7b3e-4ca3-8284-ab95ab39bb07 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:12.021073 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d421d5ee-7b3e-4ca3-8284-ab95ab39bb07 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:12.033909 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 316/651] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:43:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:12.034701 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d421d5ee-7b3e-4ca3-8284-ab95ab39bb07 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:12.044575 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3862fb7d-35ff-4c99-af60-5ce8200bf968 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:12.087210 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d682100-7bad-4fd1-b5d0-1591bcf1e466 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:12.087844 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d682100-7bad-4fd1-b5d0-1591bcf1e466 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:12.088122 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d682100-7bad-4fd1-b5d0-1591bcf1e466 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:12.088429 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d682100-7bad-4fd1-b5d0-1591bcf1e466 None admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:12.089159 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d682100-7bad-4fd1-b5d0-1591bcf1e466 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:12.089460 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d682100-7bad-4fd1-b5d0-1591bcf1e466 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:12.091009 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d682100-7bad-4fd1-b5d0-1591bcf1e466 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:12.104359 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 336/652] 10.222.0.33 () {56 vars in 1148 bytes} [Sun Oct 9 00:43:11 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:12.104988 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d682100-7bad-4fd1-b5d0-1591bcf1e466 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:12.145869 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d421d5ee-7b3e-4ca3-8284-ab95ab39bb07 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:12.165234 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:12.183809 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2048b9e8-d1ba-4fc1-b072-24aa791e19c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:12.184001 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2048b9e8-d1ba-4fc1-b072-24aa791e19c4 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:12.184285 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2048b9e8-d1ba-4fc1-b072-24aa791e19c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:12.184557 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2048b9e8-d1ba-4fc1-b072-24aa791e19c4 None admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:12.185208 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2048b9e8-d1ba-4fc1-b072-24aa791e19c4 None admin] RBAC: Adding query filter params (name=service) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:12.185510 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2048b9e8-d1ba-4fc1-b072-24aa791e19c4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:12.186862 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2048b9e8-d1ba-4fc1-b072-24aa791e19c4 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:12.200003 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 317/653] 10.222.0.33 () {56 vars in 1103 bytes} [Sun Oct 9 00:43:12 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:12.200003 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2048b9e8-d1ba-4fc1-b072-24aa791e19c4 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:12.217713 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d682100-7bad-4fd1-b5d0-1591bcf1e466 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:12.229877 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:12.256340 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f59734-3c56-4007-8be1-5e1ac8b93b0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:12.256944 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f59734-3c56-4007-8be1-5e1ac8b93b0c None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:12.257890 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f59734-3c56-4007-8be1-5e1ac8b93b0c None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:12.258165 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f59734-3c56-4007-8be1-5e1ac8b93b0c None admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:12.258925 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01f59734-3c56-4007-8be1-5e1ac8b93b0c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:12.259259 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01f59734-3c56-4007-8be1-5e1ac8b93b0c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:12.260622 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01f59734-3c56-4007-8be1-5e1ac8b93b0c None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:12.276749 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 337/654] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:43:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:12.276749 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f59734-3c56-4007-8be1-5e1ac8b93b0c None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:12.303083 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2048b9e8-d1ba-4fc1-b072-24aa791e19c4 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:12.342736 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c72a9efc-cfe4-42fc-9267-efa6087ced15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:12.343360 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c72a9efc-cfe4-42fc-9267-efa6087ced15 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:12.343659 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c72a9efc-cfe4-42fc-9267-efa6087ced15 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:12.343931 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c72a9efc-cfe4-42fc-9267-efa6087ced15 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:12.351548 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c72a9efc-cfe4-42fc-9267-efa6087ced15 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:12.352992 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c72a9efc-cfe4-42fc-9267-efa6087ced15 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:12.363996 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-c72a9efc-cfe4-42fc-9267-efa6087ced15 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Oct 09 00:43:12.363996 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:12.363996 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:12.363996 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:12.363996 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:12.363996 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:12.363996 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:12.363996 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:12.363996 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:12.363996 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:12.363996 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:12.363996 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:12.363996 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 09 00:43:12.363996 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 09 00:43:12.363996 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 09 00:43:12.363996 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 09 00:43:12.364911 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:12.364911 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:12.364911 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 09 00:43:12.364911 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 09 00:43:12.364911 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 09 00:43:12.364911 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 09 00:43:12.364911 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:12.364911 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:12.364911 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:12.364911 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:12.364911 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:12.364911 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:12.364911 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:12.364911 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:12.364911 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:12.364911 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:12.364911 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:12.364911 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:12.365826 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:12.365826 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:12.365826 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 09 00:43:12.365826 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 09 00:43:12.365826 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 09 00:43:12.365826 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 09 00:43:12.365826 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 09 00:43:12.365826 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 09 00:43:12.365826 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Oct 09 00:43:12.365826 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:12.365826 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 318/655] 10.222.0.33 () {56 vars in 1102 bytes} [Sun Oct 9 00:43:12 2022] GET /identity/v3/users/neutron => generated 85 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:12.365826 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c72a9efc-cfe4-42fc-9267-efa6087ced15 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:12.381523 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f59734-3c56-4007-8be1-5e1ac8b93b0c None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:12.410964 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:12.417965 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-371e9119-61c2-40ee-af66-0df83d83d64f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:12.418612 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-371e9119-61c2-40ee-af66-0df83d83d64f None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:12.418892 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-371e9119-61c2-40ee-af66-0df83d83d64f None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:12.419169 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-371e9119-61c2-40ee-af66-0df83d83d64f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:12.419930 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-371e9119-61c2-40ee-af66-0df83d83d64f None admin] RBAC: Adding query filter params (role.id=228232e130c5412a80d774978175d28b, scope.project.id=013c83476b1340fbb051061833c4a9c6, user.id=e0a5d95969e74157bb2d2a57207e1651) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:12.420238 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-371e9119-61c2-40ee-af66-0df83d83d64f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:12.422559 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-371e9119-61c2-40ee-af66-0df83d83d64f None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:12.428514 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 338/656] 10.222.0.33 () {56 vars in 1374 bytes} [Sun Oct 9 00:43:12 2022] GET /identity/v3/role_assignments?user.id=e0a5d95969e74157bb2d2a57207e1651&scope.project.id=013c83476b1340fbb051061833c4a9c6&role.id=228232e130c5412a80d774978175d28b => generated 608 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:12.429865 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-371e9119-61c2-40ee-af66-0df83d83d64f None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:12.482056 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:12.496743 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c72a9efc-cfe4-42fc-9267-efa6087ced15 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:12.536843 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e93646fb-b432-4a31-b368-ba1ab15a6d9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:12.537472 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e93646fb-b432-4a31-b368-ba1ab15a6d9e None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:12.537759 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e93646fb-b432-4a31-b368-ba1ab15a6d9e None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:12.538299 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e93646fb-b432-4a31-b368-ba1ab15a6d9e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:12.552713 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e93646fb-b432-4a31-b368-ba1ab15a6d9e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:12.554348 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e93646fb-b432-4a31-b368-ba1ab15a6d9e None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:12.559989 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-371e9119-61c2-40ee-af66-0df83d83d64f None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:12.574276 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-e93646fb-b432-4a31-b368-ba1ab15a6d9e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:12.574276 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:12.574276 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:12.574276 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:12.574276 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:12.574276 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:12.574276 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:12.574276 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:12.574276 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:12.574276 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:12.574276 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:12.574276 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:12.574276 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:43:12.574276 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:43:12.574276 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:43:12.574276 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:43:12.575269 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:12.575269 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:12.575269 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:12.575269 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:12.575269 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:12.575269 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:12.575269 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:12.575269 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:12.575269 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:12.575269 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:12.575269 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:12.575269 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:12.575269 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:12.575269 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:12.575269 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:12.575269 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:12.575269 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:43:12.576782 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:43:12.576782 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:12.576782 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:12.576782 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 319/657] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:43:12 2022] GET /identity/v3/domains/Default => generated 87 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:12.576782 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e93646fb-b432-4a31-b368-ba1ab15a6d9e None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:12.600893 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5ca2f63-0e5d-4895-9b47-3a8fa9f386ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:12.601631 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5ca2f63-0e5d-4895-9b47-3a8fa9f386ab None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:12.601963 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5ca2f63-0e5d-4895-9b47-3a8fa9f386ab None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:12.602243 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5ca2f63-0e5d-4895-9b47-3a8fa9f386ab None admin] PATH_INFO: `/v3/projects/service` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:12.610121 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5ca2f63-0e5d-4895-9b47-3a8fa9f386ab None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:12.611938 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5ca2f63-0e5d-4895-9b47-3a8fa9f386ab None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:12.624685 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-e5ca2f63-0e5d-4895-9b47-3a8fa9f386ab None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 09 00:43:12.624685 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:12.624685 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:12.624685 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:12.624685 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:12.624685 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:12.624685 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:12.624685 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:12.624685 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:12.624685 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:12.624685 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:12.624685 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:12.624685 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 09 00:43:12.624685 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 09 00:43:12.624685 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 09 00:43:12.625640 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 09 00:43:12.625640 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:12.625640 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:12.625640 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:12.625640 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:12.625640 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:12.625640 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:12.625640 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:12.625640 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:12.625640 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:12.625640 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:12.625640 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:12.625640 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:12.625640 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:12.625640 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:12.625640 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:12.625640 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:12.626514 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 09 00:43:12.626514 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 09 00:43:12.626514 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 09 00:43:12.626514 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 09 00:43:12.626514 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 09 00:43:12.626514 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 09 00:43:12.626514 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 09 00:43:12.626514 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:12.627747 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 339/658] 10.222.0.33 () {56 vars in 1111 bytes} [Sun Oct 9 00:43:12 2022] GET /identity/v3/projects/service => generated 88 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:12.629010 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c776f81c-6057-48ea-bbe1-4ad28a98c2c5 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:12.629289 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c776f81c-6057-48ea-bbe1-4ad28a98c2c5 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:12.629559 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c776f81c-6057-48ea-bbe1-4ad28a98c2c5 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:12.635651 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 340/659] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:43:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:43:12.635651 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c776f81c-6057-48ea-bbe1-4ad28a98c2c5 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:12.697193 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:12.708362 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e93646fb-b432-4a31-b368-ba1ab15a6d9e None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:12.742678 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:12.746192 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e8b9d7c-0209-4215-afe9-b103cad6be34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:12.747048 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e8b9d7c-0209-4215-afe9-b103cad6be34 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:12.747345 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e8b9d7c-0209-4215-afe9-b103cad6be34 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:12.747647 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e8b9d7c-0209-4215-afe9-b103cad6be34 None admin] PATH_INFO: `/v3/users` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:12.748338 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e8b9d7c-0209-4215-afe9-b103cad6be34 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:12.748647 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e8b9d7c-0209-4215-afe9-b103cad6be34 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:12.750085 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e8b9d7c-0209-4215-afe9-b103cad6be34 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:12.751821 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c776f81c-6057-48ea-bbe1-4ad28a98c2c5 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:12.773731 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 320/660] 10.222.0.33 () {56 vars in 1139 bytes} [Sun Oct 9 00:43:12 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:12.773731 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e8b9d7c-0209-4215-afe9-b103cad6be34 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:12.792525 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b019bf01-c7bd-4a75-9cd4-d1247361f473 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:12.793124 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b019bf01-c7bd-4a75-9cd4-d1247361f473 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:12.793398 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b019bf01-c7bd-4a75-9cd4-d1247361f473 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:12.793668 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b019bf01-c7bd-4a75-9cd4-d1247361f473 None admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:12.794294 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b019bf01-c7bd-4a75-9cd4-d1247361f473 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:12.794587 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b019bf01-c7bd-4a75-9cd4-d1247361f473 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:12.795985 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b019bf01-c7bd-4a75-9cd4-d1247361f473 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:12.808392 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 341/661] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:43:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:12.809637 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ee2777-860f-401a-9708-ad1e5b204cf3 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:12.809919 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ee2777-860f-401a-9708-ad1e5b204cf3 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:12.810187 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ee2777-860f-401a-9708-ad1e5b204cf3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:12.828123 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-77ee2777-860f-401a-9708-ad1e5b204cf3 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:43:12.829317 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-77ee2777-860f-401a-9708-ad1e5b204cf3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:43:12.880057 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e8b9d7c-0209-4215-afe9-b103cad6be34 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:12.900628 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 342/662] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:43:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 09 00:43:12.901490 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77ee2777-860f-401a-9708-ad1e5b204cf3 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:12.926919 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e494eb69-6ff9-4aa9-a5ba-a768c129c7d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:12.928151 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e494eb69-6ff9-4aa9-a5ba-a768c129c7d9 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:12.928650 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e494eb69-6ff9-4aa9-a5ba-a768c129c7d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:12.930010 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e494eb69-6ff9-4aa9-a5ba-a768c129c7d9 None admin] PATH_INFO: `/v3/projects` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:12.931187 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e494eb69-6ff9-4aa9-a5ba-a768c129c7d9 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:12.931812 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e494eb69-6ff9-4aa9-a5ba-a768c129c7d9 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:12.933628 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e494eb69-6ff9-4aa9-a5ba-a768c129c7d9 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:12.953664 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 321/663] 10.222.0.33 () {56 vars in 1148 bytes} [Sun Oct 9 00:43:12 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:12.953664 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e494eb69-6ff9-4aa9-a5ba-a768c129c7d9 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:12.966521 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:13.015369 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:13.041100 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77ee2777-860f-401a-9708-ad1e5b204cf3 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:13.076406 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e494eb69-6ff9-4aa9-a5ba-a768c129c7d9 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:13.085611 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d94d06a-8e65-4f59-9427-4e034dec47da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:13.086451 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d94d06a-8e65-4f59-9427-4e034dec47da None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:13.086930 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d94d06a-8e65-4f59-9427-4e034dec47da None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:13.087505 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d94d06a-8e65-4f59-9427-4e034dec47da None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:13.121583 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db26d9e5-5184-4507-8574-a295e3944633 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:13.122247 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db26d9e5-5184-4507-8574-a295e3944633 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:13.122531 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db26d9e5-5184-4507-8574-a295e3944633 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:13.122810 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db26d9e5-5184-4507-8574-a295e3944633 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:13.131921 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db26d9e5-5184-4507-8574-a295e3944633 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:13.134424 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db26d9e5-5184-4507-8574-a295e3944633 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:13.148059 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-db26d9e5-5184-4507-8574-a295e3944633 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Oct 09 00:43:13.148059 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:13.148059 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:13.148059 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:13.148059 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:13.148059 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:13.148059 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:13.148059 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:13.148059 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:13.148059 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:13.148059 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:13.148059 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:13.148059 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 09 00:43:13.148059 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 09 00:43:13.148059 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 09 00:43:13.148059 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 09 00:43:13.148979 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:13.148979 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:13.148979 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 09 00:43:13.148979 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 09 00:43:13.148979 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 09 00:43:13.148979 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 09 00:43:13.148979 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:13.148979 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:13.148979 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:13.148979 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:13.148979 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:13.148979 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:13.148979 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:13.148979 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:13.148979 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:13.148979 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:13.148979 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:13.148979 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:13.149915 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:13.149915 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:13.149915 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 09 00:43:13.149915 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 09 00:43:13.149915 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 09 00:43:13.149915 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 09 00:43:13.149915 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 09 00:43:13.149915 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 09 00:43:13.149915 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Oct 09 00:43:13.149915 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:13.149915 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 322/664] 10.222.0.33 () {56 vars in 1099 bytes} [Sun Oct 9 00:43:12 2022] GET /identity/v3/users/cinder => generated 84 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:13.150507 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db26d9e5-5184-4507-8574-a295e3944633 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:13.167234 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d94d06a-8e65-4f59-9427-4e034dec47da None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:13.169270 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d94d06a-8e65-4f59-9427-4e034dec47da None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:13.224093 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:13.257723 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db26d9e5-5184-4507-8574-a295e3944633 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:13.260491 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:13.281136 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 343/665] 10.222.0.33 () {58 vars in 1282 bytes} [Sun Oct 9 00:43:12 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 380 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 09 00:43:13.281888 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d94d06a-8e65-4f59-9427-4e034dec47da None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:13.298625 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24ddaecd-10bd-4aca-b957-50cbc251adce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:13.299524 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24ddaecd-10bd-4aca-b957-50cbc251adce None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:13.300057 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24ddaecd-10bd-4aca-b957-50cbc251adce None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:13.300641 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24ddaecd-10bd-4aca-b957-50cbc251adce None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:13.301686 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24ddaecd-10bd-4aca-b957-50cbc251adce None admin] RBAC: Adding query filter params (role.id=228232e130c5412a80d774978175d28b, scope.project.id=013c83476b1340fbb051061833c4a9c6, user.id=1f7a88094d73471787cc0a287adbd394) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:13.302219 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24ddaecd-10bd-4aca-b957-50cbc251adce None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:13.303995 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24ddaecd-10bd-4aca-b957-50cbc251adce None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:13.313058 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 323/666] 10.222.0.33 () {56 vars in 1374 bytes} [Sun Oct 9 00:43:13 2022] GET /identity/v3/role_assignments?user.id=1f7a88094d73471787cc0a287adbd394&scope.project.id=013c83476b1340fbb051061833c4a9c6&role.id=228232e130c5412a80d774978175d28b => generated 608 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:13.313058 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24ddaecd-10bd-4aca-b957-50cbc251adce None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:13.402271 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d94d06a-8e65-4f59-9427-4e034dec47da None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:13.431778 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24ddaecd-10bd-4aca-b957-50cbc251adce None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:13.445081 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16ca36fb-7bbb-4f37-9346-08b0feacd499 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:13.446099 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16ca36fb-7bbb-4f37-9346-08b0feacd499 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:13.446609 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16ca36fb-7bbb-4f37-9346-08b0feacd499 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:13.447195 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16ca36fb-7bbb-4f37-9346-08b0feacd499 None admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:13.448298 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16ca36fb-7bbb-4f37-9346-08b0feacd499 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:13.448830 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16ca36fb-7bbb-4f37-9346-08b0feacd499 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:13.450519 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16ca36fb-7bbb-4f37-9346-08b0feacd499 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:13.471470 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51eb179d-f1c6-4064-81b6-8d170f6c73de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:13.472228 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51eb179d-f1c6-4064-81b6-8d170f6c73de None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:13.472511 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51eb179d-f1c6-4064-81b6-8d170f6c73de None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:13.472783 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51eb179d-f1c6-4064-81b6-8d170f6c73de None admin] PATH_INFO: `/v3/roles/service` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:13.477839 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 344/667] 10.222.0.33 () {56 vars in 1137 bytes} [Sun Oct 9 00:43:13 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:13.477839 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16ca36fb-7bbb-4f37-9346-08b0feacd499 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:13.479926 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:13.495097 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51eb179d-f1c6-4064-81b6-8d170f6c73de None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:43:13.501929 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-51eb179d-f1c6-4064-81b6-8d170f6c73de None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 09 00:43:13.501929 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:13.501929 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:13.501929 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:13.501929 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:13.501929 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:13.501929 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:13.501929 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:13.501929 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:13.501929 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:13.501929 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:13.501929 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:13.501929 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:43:13.501929 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:43:13.501929 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:43:13.501929 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:43:13.503241 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:43:13.503241 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:43:13.503241 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:43:13.503241 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:43:13.503241 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:13.503241 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:13.503241 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:13.503241 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:13.503241 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:13.503241 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:13.503241 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:13.503241 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:13.503241 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:13.503241 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:13.503241 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:13.503241 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:13.504425 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:13.504425 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:13.504425 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:13.504425 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:13.504425 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:43:13.504425 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:43:13.504425 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:43:13.504425 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:43:13.504425 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:43:13.504425 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:43:13.504425 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 09 00:43:13.504425 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:13.507711 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 324/668] 10.222.0.33 () {56 vars in 1102 bytes} [Sun Oct 9 00:43:13 2022] GET /identity/v3/roles/service => generated 85 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:13.508762 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51eb179d-f1c6-4064-81b6-8d170f6c73de None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:13.542566 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:13.614219 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16ca36fb-7bbb-4f37-9346-08b0feacd499 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:13.633187 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51eb179d-f1c6-4064-81b6-8d170f6c73de None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:13.659729 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb339708-c687-4583-a1eb-f567ee0b86b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:13.660372 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb339708-c687-4583-a1eb-f567ee0b86b1 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:13.660653 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb339708-c687-4583-a1eb-f567ee0b86b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:13.660930 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb339708-c687-4583-a1eb-f567ee0b86b1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:13.669475 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3683b73-1f7c-4c73-84b6-c50ddfde3b62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:13.670333 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3683b73-1f7c-4c73-84b6-c50ddfde3b62 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:13.670831 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3683b73-1f7c-4c73-84b6-c50ddfde3b62 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:13.671349 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3683b73-1f7c-4c73-84b6-c50ddfde3b62 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:13.673849 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb339708-c687-4583-a1eb-f567ee0b86b1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:13.676282 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb339708-c687-4583-a1eb-f567ee0b86b1 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:13.699227 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-eb339708-c687-4583-a1eb-f567ee0b86b1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:13.699227 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:13.699227 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:13.699227 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:13.699227 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:13.699227 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:13.699227 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:13.699227 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:13.699227 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:13.699227 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:13.699227 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:13.699227 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:13.699227 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:43:13.699227 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:43:13.699227 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:43:13.699227 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:43:13.700214 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:13.700214 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:13.700214 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:13.700214 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:13.700214 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:13.700214 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:13.700214 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:13.700214 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:13.700214 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:13.700214 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:13.700214 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:13.700214 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:13.700214 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:13.700214 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:13.700214 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:13.700214 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:13.700214 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:43:13.701117 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:43:13.701117 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:13.701117 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:13.703178 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 345/669] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:43:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 227 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:13.708747 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb339708-c687-4583-a1eb-f567ee0b86b1 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:13.742247 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:13.750769 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3683b73-1f7c-4c73-84b6-c50ddfde3b62 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:13.752752 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3683b73-1f7c-4c73-84b6-c50ddfde3b62 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:13.835740 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:13.845538 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb339708-c687-4583-a1eb-f567ee0b86b1 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:13.877123 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 325/670] 10.222.0.33 () {58 vars in 1282 bytes} [Sun Oct 9 00:43:13 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 365 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 09 00:43:13.877123 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3683b73-1f7c-4c73-84b6-c50ddfde3b62 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:13.895085 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02ba4c9b-ca14-4113-aa7a-4529410bd424 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:13.896550 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ba4c9b-ca14-4113-aa7a-4529410bd424 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:13.897072 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ba4c9b-ca14-4113-aa7a-4529410bd424 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:13.897543 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ba4c9b-ca14-4113-aa7a-4529410bd424 None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:13.898569 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02ba4c9b-ca14-4113-aa7a-4529410bd424 None admin] RBAC: Adding query filter params (name=service) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:13.899069 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02ba4c9b-ca14-4113-aa7a-4529410bd424 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:13.901469 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02ba4c9b-ca14-4113-aa7a-4529410bd424 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:13.913863 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 346/671] 10.222.0.33 () {56 vars in 1103 bytes} [Sun Oct 9 00:43:13 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:13.915057 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02ba4c9b-ca14-4113-aa7a-4529410bd424 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:13.991371 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:13.999681 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3683b73-1f7c-4c73-84b6-c50ddfde3b62 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:14.040765 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3f2a73e-9c41-4409-befc-046fc451773b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:14.041381 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3f2a73e-9c41-4409-befc-046fc451773b None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:14.041922 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3f2a73e-9c41-4409-befc-046fc451773b None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:14.042204 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3f2a73e-9c41-4409-befc-046fc451773b None admin] PATH_INFO: `/v3/domains` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:14.042838 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3f2a73e-9c41-4409-befc-046fc451773b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:14.043146 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3f2a73e-9c41-4409-befc-046fc451773b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:14.044523 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3f2a73e-9c41-4409-befc-046fc451773b None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:14.056617 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02ba4c9b-ca14-4113-aa7a-4529410bd424 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:14.062287 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 326/672] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:43:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:14.062287 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3f2a73e-9c41-4409-befc-046fc451773b None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:14.108061 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78247e85-67a8-4e58-beeb-29bcb727df39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:14.108619 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78247e85-67a8-4e58-beeb-29bcb727df39 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:14.109026 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78247e85-67a8-4e58-beeb-29bcb727df39 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:14.109309 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78247e85-67a8-4e58-beeb-29bcb727df39 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:14.120901 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78247e85-67a8-4e58-beeb-29bcb727df39 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:14.122355 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78247e85-67a8-4e58-beeb-29bcb727df39 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:14.137248 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-78247e85-67a8-4e58-beeb-29bcb727df39 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:14.137248 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:14.137248 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:14.137248 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:14.137248 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:14.137248 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:14.137248 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:14.137248 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:14.137248 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:14.137248 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:14.137248 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:14.137248 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:14.137248 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:43:14.137248 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:43:14.137248 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:43:14.137248 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:43:14.138294 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:14.138294 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:14.138294 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:14.138294 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:14.138294 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:14.138294 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:14.138294 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:14.138294 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:14.138294 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:14.138294 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:14.138294 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:14.138294 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:14.138294 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:14.138294 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:14.138294 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:14.138294 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:14.138294 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:43:14.139205 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:43:14.139205 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:14.139205 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:14.142971 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 347/673] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:43:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 224 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:14.143866 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78247e85-67a8-4e58-beeb-29bcb727df39 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:14.155464 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:14.191728 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3f2a73e-9c41-4409-befc-046fc451773b None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:14.232290 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b915a623-5392-46f3-8c61-0200aa94abaa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:14.233109 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b915a623-5392-46f3-8c61-0200aa94abaa None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:14.233485 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b915a623-5392-46f3-8c61-0200aa94abaa None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:14.233977 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b915a623-5392-46f3-8c61-0200aa94abaa None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:14.249005 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b915a623-5392-46f3-8c61-0200aa94abaa None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:14.250597 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b915a623-5392-46f3-8c61-0200aa94abaa None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:14.252170 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78247e85-67a8-4e58-beeb-29bcb727df39 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:14.269208 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-b915a623-5392-46f3-8c61-0200aa94abaa None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:14.269208 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:14.269208 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:14.269208 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:14.269208 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:14.269208 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:14.269208 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:14.269208 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:14.269208 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:14.269208 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:14.269208 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:14.269208 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:14.269208 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:43:14.269208 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:43:14.269208 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:43:14.269208 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:43:14.270382 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:14.270382 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:14.270382 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:14.270382 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:14.270382 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:14.270382 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:14.270382 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:14.270382 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:14.270382 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:14.270382 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:14.270382 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:14.270382 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:14.270382 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:14.270382 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:14.270382 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:14.270382 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:14.270382 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:43:14.271421 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:43:14.271421 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:14.271421 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:14.277838 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 327/674] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:43:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:14.277838 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b915a623-5392-46f3-8c61-0200aa94abaa None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:14.290327 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:14.299255 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a4f3505-8003-4d47-a42c-97097fb3c354 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:14.300387 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a4f3505-8003-4d47-a42c-97097fb3c354 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:14.300673 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a4f3505-8003-4d47-a42c-97097fb3c354 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:14.300942 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a4f3505-8003-4d47-a42c-97097fb3c354 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:14.310737 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a4f3505-8003-4d47-a42c-97097fb3c354 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:14.312227 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a4f3505-8003-4d47-a42c-97097fb3c354 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:14.324245 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-1a4f3505-8003-4d47-a42c-97097fb3c354 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 09 00:43:14.324245 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:14.324245 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:14.324245 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:14.324245 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:14.324245 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:14.324245 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:14.324245 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:14.324245 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:14.324245 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:14.324245 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:14.324245 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:14.324245 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 09 00:43:14.324245 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 09 00:43:14.324245 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 09 00:43:14.325129 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 09 00:43:14.325129 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:14.325129 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:14.325129 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:14.325129 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:14.325129 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:14.325129 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:14.325129 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:14.325129 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:14.325129 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:14.325129 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:14.325129 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:14.325129 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:14.325129 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:14.325129 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:14.325129 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:14.325129 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:14.326012 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 09 00:43:14.326012 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 09 00:43:14.326012 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 09 00:43:14.326012 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 09 00:43:14.326012 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 09 00:43:14.326012 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 09 00:43:14.326012 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 09 00:43:14.326012 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:14.330136 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 348/675] 10.222.0.33 () {56 vars in 1111 bytes} [Sun Oct 9 00:43:14 2022] GET /identity/v3/projects/service => generated 88 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:14.330978 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a4f3505-8003-4d47-a42c-97097fb3c354 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:14.402546 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:14.420507 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b915a623-5392-46f3-8c61-0200aa94abaa None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:14.441801 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a4f3505-8003-4d47-a42c-97097fb3c354 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:14.467856 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de388637-342b-403c-9fa9-a758da2b9432 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:14.469076 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de388637-342b-403c-9fa9-a758da2b9432 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:14.469821 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de388637-342b-403c-9fa9-a758da2b9432 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:14.470752 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de388637-342b-403c-9fa9-a758da2b9432 None admin] PATH_INFO: `/v3/domains` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:14.471893 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de388637-342b-403c-9fa9-a758da2b9432 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:14.472641 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de388637-342b-403c-9fa9-a758da2b9432 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:14.474918 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de388637-342b-403c-9fa9-a758da2b9432 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:14.478106 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e920607d-4b2d-4ba8-ac4b-8c628bd8aede None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:14.478711 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e920607d-4b2d-4ba8-ac4b-8c628bd8aede None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:14.478991 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e920607d-4b2d-4ba8-ac4b-8c628bd8aede None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:14.479267 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e920607d-4b2d-4ba8-ac4b-8c628bd8aede None admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:14.479931 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e920607d-4b2d-4ba8-ac4b-8c628bd8aede None admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:14.480246 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e920607d-4b2d-4ba8-ac4b-8c628bd8aede None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:14.481596 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e920607d-4b2d-4ba8-ac4b-8c628bd8aede None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:14.491009 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 328/676] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:43:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:14.492695 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de388637-342b-403c-9fa9-a758da2b9432 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:14.498576 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 349/677] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:43:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:14.499760 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e920607d-4b2d-4ba8-ac4b-8c628bd8aede None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:14.589888 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:14.612409 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e920607d-4b2d-4ba8-ac4b-8c628bd8aede None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:14.637560 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de388637-342b-403c-9fa9-a758da2b9432 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:14.650655 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:14.662268 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e12b9f0b-c9ff-4638-95ba-809f7d70ad54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:14.664207 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e12b9f0b-c9ff-4638-95ba-809f7d70ad54 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:14.665086 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e12b9f0b-c9ff-4638-95ba-809f7d70ad54 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:14.665672 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e12b9f0b-c9ff-4638-95ba-809f7d70ad54 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:14.674828 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e12b9f0b-c9ff-4638-95ba-809f7d70ad54 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:14.675847 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-641bc8a9-22c5-42a2-9def-1e5a6d7385ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:14.676557 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641bc8a9-22c5-42a2-9def-1e5a6d7385ce None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:14.676912 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641bc8a9-22c5-42a2-9def-1e5a6d7385ce None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:14.677132 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e12b9f0b-c9ff-4638-95ba-809f7d70ad54 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:14.677276 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641bc8a9-22c5-42a2-9def-1e5a6d7385ce None admin] PATH_INFO: `/v3/projects` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:14.678186 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-641bc8a9-22c5-42a2-9def-1e5a6d7385ce None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:14.678634 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-641bc8a9-22c5-42a2-9def-1e5a6d7385ce None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:14.680284 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-641bc8a9-22c5-42a2-9def-1e5a6d7385ce None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:14.692125 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-e12b9f0b-c9ff-4638-95ba-809f7d70ad54 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 09 00:43:14.692125 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:14.692125 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:14.692125 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:14.692125 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:14.692125 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:14.692125 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:14.692125 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:14.692125 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:14.692125 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:14.692125 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:14.692125 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:14.692125 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 09 00:43:14.692125 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 09 00:43:14.692125 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 09 00:43:14.693044 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 09 00:43:14.693044 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:14.693044 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:14.693044 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:14.693044 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:14.693044 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:14.693044 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:14.693044 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:14.693044 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:14.693044 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:14.693044 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:14.693044 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:14.693044 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:14.693044 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:14.693044 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:14.693044 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:14.693044 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:14.693916 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 09 00:43:14.693916 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 09 00:43:14.693916 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 09 00:43:14.693916 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 09 00:43:14.693916 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 09 00:43:14.693916 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 09 00:43:14.693916 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 09 00:43:14.693916 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:14.701005 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 329/679] 10.222.0.33 () {56 vars in 1148 bytes} [Sun Oct 9 00:43:14 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:14.701005 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-641bc8a9-22c5-42a2-9def-1e5a6d7385ce None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:14.701343 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 350/679] 10.222.0.33 () {56 vars in 1111 bytes} [Sun Oct 9 00:43:14 2022] GET /identity/v3/projects/service => generated 88 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:14.702682 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e12b9f0b-c9ff-4638-95ba-809f7d70ad54 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:14.803019 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-641bc8a9-22c5-42a2-9def-1e5a6d7385ce None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:14.813006 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e12b9f0b-c9ff-4638-95ba-809f7d70ad54 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:14.829195 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:14.838793 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b88e67f0-35d4-4bdd-acd2-964bf91492bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:14.839782 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88e67f0-35d4-4bdd-acd2-964bf91492bd None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:14.840216 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88e67f0-35d4-4bdd-acd2-964bf91492bd None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:14.840492 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88e67f0-35d4-4bdd-acd2-964bf91492bd None admin] PATH_INFO: `/v3/users/swift` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:14.846564 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2981bce-2d23-421a-928b-84ff0c7708b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:14.847159 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2981bce-2d23-421a-928b-84ff0c7708b3 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:14.847439 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2981bce-2d23-421a-928b-84ff0c7708b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:14.847743 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2981bce-2d23-421a-928b-84ff0c7708b3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:14.848094 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b88e67f0-35d4-4bdd-acd2-964bf91492bd None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:14.848461 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2981bce-2d23-421a-928b-84ff0c7708b3 None admin] RBAC: Adding query filter params (role.id=228232e130c5412a80d774978175d28b, scope.project.id=013c83476b1340fbb051061833c4a9c6, user.id=7d2eabf9865847ceb6e44d55126078a8) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:14.848769 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2981bce-2d23-421a-928b-84ff0c7708b3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:14.849522 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b88e67f0-35d4-4bdd-acd2-964bf91492bd None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:14.850175 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2981bce-2d23-421a-928b-84ff0c7708b3 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:14.863631 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 351/680] 10.222.0.33 () {56 vars in 1374 bytes} [Sun Oct 9 00:43:14 2022] GET /identity/v3/role_assignments?user.id=7d2eabf9865847ceb6e44d55126078a8&scope.project.id=013c83476b1340fbb051061833c4a9c6&role.id=228232e130c5412a80d774978175d28b => generated 608 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:14.863631 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2981bce-2d23-421a-928b-84ff0c7708b3 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:14.863919 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-b88e67f0-35d4-4bdd-acd2-964bf91492bd None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 09 00:43:14.863919 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:14.863919 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:14.863919 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:14.863919 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:14.863919 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:14.863919 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:14.863919 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:14.863919 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:14.863919 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:14.863919 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:14.863919 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:14.863919 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 09 00:43:14.863919 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 09 00:43:14.863919 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 09 00:43:14.863919 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 09 00:43:14.864836 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:14.864836 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:14.864836 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 09 00:43:14.864836 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 09 00:43:14.864836 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 09 00:43:14.864836 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 09 00:43:14.864836 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:14.864836 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:14.864836 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:14.864836 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:14.864836 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:14.864836 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:14.864836 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:14.864836 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:14.864836 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:14.864836 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:14.864836 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:14.864836 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:14.865746 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:14.865746 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:14.865746 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 09 00:43:14.865746 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 09 00:43:14.865746 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 09 00:43:14.865746 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 09 00:43:14.865746 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 09 00:43:14.865746 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 09 00:43:14.865746 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 09 00:43:14.865746 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:14.865746 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 330/681] 10.222.0.33 () {56 vars in 1096 bytes} [Sun Oct 9 00:43:14 2022] GET /identity/v3/users/swift => generated 83 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:14.865746 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e48ac37-caa8-44ce-a26c-822d0e9db131 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:14.865746 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e48ac37-caa8-44ce-a26c-822d0e9db131 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:14.867051 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e48ac37-caa8-44ce-a26c-822d0e9db131 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:14.867051 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 331/682] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:43:14 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:43:14.872169 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e48ac37-caa8-44ce-a26c-822d0e9db131 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:14.939905 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:14.999331 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e48ac37-caa8-44ce-a26c-822d0e9db131 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:15.000204 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2981bce-2d23-421a-928b-84ff0c7708b3 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:15.040449 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-341d50da-1a6f-4876-9451-263549cee5c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:15.040681 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-341d50da-1a6f-4876-9451-263549cee5c7 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:15.040763 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-341d50da-1a6f-4876-9451-263549cee5c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:15.041086 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-341d50da-1a6f-4876-9451-263549cee5c7 None admin] PATH_INFO: `/v3/users` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:15.041910 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-341d50da-1a6f-4876-9451-263549cee5c7 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:15.042086 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-341d50da-1a6f-4876-9451-263549cee5c7 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:15.043672 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-341d50da-1a6f-4876-9451-263549cee5c7 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:15.048004 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-160eddac-3d1b-4012-a71f-c204a1c07548 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:15.048795 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-160eddac-3d1b-4012-a71f-c204a1c07548 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:15.049246 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-160eddac-3d1b-4012-a71f-c204a1c07548 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:15.049678 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-160eddac-3d1b-4012-a71f-c204a1c07548 None admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:15.050371 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-160eddac-3d1b-4012-a71f-c204a1c07548 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:15.050843 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-160eddac-3d1b-4012-a71f-c204a1c07548 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:15.052390 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-160eddac-3d1b-4012-a71f-c204a1c07548 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:15.062842 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 332/683] 10.222.0.33 () {56 vars in 1135 bytes} [Sun Oct 9 00:43:14 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:15.064160 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-552db7a0-c5fd-4b3d-bcf6-ec4e33fbee06 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:15.064447 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-552db7a0-c5fd-4b3d-bcf6-ec4e33fbee06 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:15.064722 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-552db7a0-c5fd-4b3d-bcf6-ec4e33fbee06 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:15.070155 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 352/684] 10.222.0.33 () {56 vars in 1148 bytes} [Sun Oct 9 00:43:14 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:15.070997 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-160eddac-3d1b-4012-a71f-c204a1c07548 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:15.091668 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-552db7a0-c5fd-4b3d-bcf6-ec4e33fbee06 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:43:15.092642 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-552db7a0-c5fd-4b3d-bcf6-ec4e33fbee06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:43:15.139028 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:15.180508 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 333/685] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:43:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 09 00:43:15.181692 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-552db7a0-c5fd-4b3d-bcf6-ec4e33fbee06 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:15.195976 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-160eddac-3d1b-4012-a71f-c204a1c07548 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:15.234909 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dc5b8e1-ef39-4b8c-803a-002c2e399922 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:15.235527 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc5b8e1-ef39-4b8c-803a-002c2e399922 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:15.235837 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc5b8e1-ef39-4b8c-803a-002c2e399922 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:15.236115 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc5b8e1-ef39-4b8c-803a-002c2e399922 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:15.271985 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:15.311783 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dc5b8e1-ef39-4b8c-803a-002c2e399922 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:15.313300 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dc5b8e1-ef39-4b8c-803a-002c2e399922 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:15.406367 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-552db7a0-c5fd-4b3d-bcf6-ec4e33fbee06 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:15.421886 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 353/686] 10.222.0.33 () {58 vars in 1282 bytes} [Sun Oct 9 00:43:15 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 351 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 09 00:43:15.428096 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dc5b8e1-ef39-4b8c-803a-002c2e399922 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:15.453905 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:15.478365 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeeec94f-873e-46e6-91b5-21a1821bbabd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:15.479022 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeeec94f-873e-46e6-91b5-21a1821bbabd None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:15.479304 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeeec94f-873e-46e6-91b5-21a1821bbabd None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:15.479577 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeeec94f-873e-46e6-91b5-21a1821bbabd None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:15.480328 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeeec94f-873e-46e6-91b5-21a1821bbabd None admin] RBAC: Adding query filter params (role.id=228232e130c5412a80d774978175d28b, scope.project.id=013c83476b1340fbb051061833c4a9c6, user.id=f3f48335668a4e8f934ac11ff66cd308) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:15.480636 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeeec94f-873e-46e6-91b5-21a1821bbabd None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:15.482098 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeeec94f-873e-46e6-91b5-21a1821bbabd None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:15.497263 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 334/687] 10.222.0.33 () {56 vars in 1374 bytes} [Sun Oct 9 00:43:15 2022] GET /identity/v3/role_assignments?user.id=f3f48335668a4e8f934ac11ff66cd308&scope.project.id=013c83476b1340fbb051061833c4a9c6&role.id=228232e130c5412a80d774978175d28b => generated 608 bytes in 316 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:15.583255 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dc5b8e1-ef39-4b8c-803a-002c2e399922 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:15.602046 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeeec94f-873e-46e6-91b5-21a1821bbabd None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:15.632749 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fc6cd63-c6fb-4f15-98bf-e886e7c5e65f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:15.633392 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc6cd63-c6fb-4f15-98bf-e886e7c5e65f None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:15.633986 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc6cd63-c6fb-4f15-98bf-e886e7c5e65f None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:15.634097 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc6cd63-c6fb-4f15-98bf-e886e7c5e65f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:15.646441 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:15.650484 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fc6cd63-c6fb-4f15-98bf-e886e7c5e65f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:15.652216 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fc6cd63-c6fb-4f15-98bf-e886e7c5e65f None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:15.676169 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-5fc6cd63-c6fb-4f15-98bf-e886e7c5e65f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:15.676169 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:15.676169 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:15.676169 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:15.676169 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:15.676169 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:15.676169 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:15.676169 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:15.676169 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:15.676169 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:15.676169 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:15.676169 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:15.676169 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:43:15.676169 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:43:15.676169 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:43:15.676169 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:43:15.677165 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:15.677165 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:15.677165 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:15.677165 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:15.677165 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:15.677165 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:15.677165 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:15.677165 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:15.677165 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:15.677165 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:15.677165 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:15.677165 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:15.677165 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:15.677165 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:15.677165 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:15.677165 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:15.677165 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:43:15.678032 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:43:15.678032 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:15.678032 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:15.678032 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 354/688] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:43:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 250 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:15.678838 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e8858c-4a15-4c27-8247-f880c8527d9d None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:15.679158 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e8858c-4a15-4c27-8247-f880c8527d9d None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:15.679434 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e8858c-4a15-4c27-8247-f880c8527d9d None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:15.680983 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 355/689] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:43:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:43:15.689966 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3e8858c-4a15-4c27-8247-f880c8527d9d None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:15.724743 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeeec94f-873e-46e6-91b5-21a1821bbabd None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:15.769497 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0da38db9-56b5-4fa0-9ceb-458d4bbdf7ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:15.770292 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da38db9-56b5-4fa0-9ceb-458d4bbdf7ee None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:15.770645 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da38db9-56b5-4fa0-9ceb-458d4bbdf7ee None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:15.770985 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da38db9-56b5-4fa0-9ceb-458d4bbdf7ee None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:15.780589 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:15.785686 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0da38db9-56b5-4fa0-9ceb-458d4bbdf7ee None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:43:15.792664 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-0da38db9-56b5-4fa0-9ceb-458d4bbdf7ee None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 09 00:43:15.792664 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:15.792664 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:15.792664 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:15.792664 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:15.792664 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:15.792664 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:15.792664 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:15.792664 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:15.792664 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:15.792664 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:15.792664 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:15.792664 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:43:15.792664 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:43:15.792664 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:43:15.792664 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:43:15.793585 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:43:15.793585 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:43:15.793585 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:43:15.793585 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:43:15.793585 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:15.793585 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:15.793585 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:15.793585 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:15.793585 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:15.793585 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:15.793585 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:15.793585 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:15.793585 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:15.793585 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:15.793585 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:15.793585 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:15.794401 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:15.794401 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:15.794401 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:15.794401 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:15.794401 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:43:15.794401 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:43:15.794401 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:43:15.794401 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:43:15.794401 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:43:15.794401 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:43:15.794401 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 09 00:43:15.794401 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:15.794401 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 335/690] 10.222.0.33 () {56 vars in 1096 bytes} [Sun Oct 9 00:43:15 2022] GET /identity/v3/roles/admin => generated 83 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 09 00:43:15.795092 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07bf65a3-6c5b-40a2-b542-b096afb9f901 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:15.795678 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07bf65a3-6c5b-40a2-b542-b096afb9f901 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:15.795678 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07bf65a3-6c5b-40a2-b542-b096afb9f901 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:15.819063 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-07bf65a3-6c5b-40a2-b542-b096afb9f901 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:43:15.820157 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-07bf65a3-6c5b-40a2-b542-b096afb9f901 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:43:15.835891 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3e8858c-4a15-4c27-8247-f880c8527d9d None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:15.881572 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62d80bc5-0aef-4c39-af85-3992475be9c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:15.882803 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d80bc5-0aef-4c39-af85-3992475be9c7 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:15.883091 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d80bc5-0aef-4c39-af85-3992475be9c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:15.883369 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d80bc5-0aef-4c39-af85-3992475be9c7 None admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:15.884080 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62d80bc5-0aef-4c39-af85-3992475be9c7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:15.884384 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62d80bc5-0aef-4c39-af85-3992475be9c7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:15.885758 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62d80bc5-0aef-4c39-af85-3992475be9c7 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:15.895546 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 336/691] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:43:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 09 00:43:15.896783 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07bf65a3-6c5b-40a2-b542-b096afb9f901 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:15.898594 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 356/692] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:43:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:15.903953 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62d80bc5-0aef-4c39-af85-3992475be9c7 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:15.947261 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:16.066503 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62d80bc5-0aef-4c39-af85-3992475be9c7 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:16.077197 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07bf65a3-6c5b-40a2-b542-b096afb9f901 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:16.110128 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:16.131208 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25349461-ae98-4988-a136-ee4f825efe34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:16.131855 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25349461-ae98-4988-a136-ee4f825efe34 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:16.132701 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25349461-ae98-4988-a136-ee4f825efe34 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:16.132979 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25349461-ae98-4988-a136-ee4f825efe34 None admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:16.133638 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25349461-ae98-4988-a136-ee4f825efe34 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:16.134096 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25349461-ae98-4988-a136-ee4f825efe34 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:16.135532 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25349461-ae98-4988-a136-ee4f825efe34 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:16.144294 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7272323d-e423-4ad9-8718-1509ea67f523 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:16.144450 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 337/693] 10.222.0.33 () {56 vars in 1099 bytes} [Sun Oct 9 00:43:15 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 248 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:16.144936 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7272323d-e423-4ad9-8718-1509ea67f523 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:16.145218 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7272323d-e423-4ad9-8718-1509ea67f523 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:16.145612 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7272323d-e423-4ad9-8718-1509ea67f523 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:16.149916 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25349461-ae98-4988-a136-ee4f825efe34 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:16.155802 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7272323d-e423-4ad9-8718-1509ea67f523 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:16.157288 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7272323d-e423-4ad9-8718-1509ea67f523 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:16.181974 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-7272323d-e423-4ad9-8718-1509ea67f523 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 09 00:43:16.181974 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:16.181974 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:16.181974 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:16.181974 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:16.181974 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:16.181974 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:16.181974 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:16.181974 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:16.181974 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:16.181974 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:16.181974 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:16.181974 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 09 00:43:16.181974 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 09 00:43:16.181974 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 09 00:43:16.182900 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 09 00:43:16.182900 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:16.182900 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:16.182900 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:16.182900 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:16.182900 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:16.182900 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:16.182900 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:16.182900 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:16.182900 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:16.182900 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:16.182900 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:16.182900 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:16.182900 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:16.182900 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:16.182900 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:16.182900 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:16.183820 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 09 00:43:16.183820 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 09 00:43:16.183820 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 09 00:43:16.183820 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 09 00:43:16.183820 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 09 00:43:16.183820 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 09 00:43:16.183820 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 09 00:43:16.183820 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:16.183820 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 357/694] 10.222.0.33 () {56 vars in 1111 bytes} [Sun Oct 9 00:43:15 2022] GET /identity/v3/projects/service => generated 88 bytes in 280 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 09 00:43:16.188857 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7272323d-e423-4ad9-8718-1509ea67f523 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:16.259554 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:16.270810 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25349461-ae98-4988-a136-ee4f825efe34 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:16.307620 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09ec3749-0547-4b36-bfb7-dca8eee5bde2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:16.308276 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ec3749-0547-4b36-bfb7-dca8eee5bde2 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:16.308628 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ec3749-0547-4b36-bfb7-dca8eee5bde2 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:16.308905 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ec3749-0547-4b36-bfb7-dca8eee5bde2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:16.322285 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09ec3749-0547-4b36-bfb7-dca8eee5bde2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:16.323962 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09ec3749-0547-4b36-bfb7-dca8eee5bde2 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:16.328170 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7272323d-e423-4ad9-8718-1509ea67f523 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:16.340293 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-09ec3749-0547-4b36-bfb7-dca8eee5bde2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:16.340293 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:16.340293 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:16.340293 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:16.340293 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:16.340293 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:16.340293 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:16.340293 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:16.340293 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:16.340293 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:16.340293 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:16.340293 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:16.340293 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:43:16.340293 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:43:16.340293 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:43:16.340293 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:43:16.341313 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:16.341313 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:16.341313 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:16.341313 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:16.341313 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:16.341313 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:16.341313 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:16.341313 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:16.341313 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:16.341313 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:16.341313 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:16.341313 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:16.341313 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:16.341313 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:16.341313 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:16.341313 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:16.341313 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:43:16.342203 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:43:16.342203 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:16.342203 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:16.342203 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 338/695] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:43:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 09 00:43:16.343411 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09ec3749-0547-4b36-bfb7-dca8eee5bde2 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:16.374174 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbd73dda-606d-427d-b082-97e9b95d2388 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:16.375383 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd73dda-606d-427d-b082-97e9b95d2388 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:16.375875 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd73dda-606d-427d-b082-97e9b95d2388 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:16.376582 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd73dda-606d-427d-b082-97e9b95d2388 None admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:16.377866 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbd73dda-606d-427d-b082-97e9b95d2388 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:16.378597 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbd73dda-606d-427d-b082-97e9b95d2388 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:16.380527 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbd73dda-606d-427d-b082-97e9b95d2388 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:16.403065 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 358/696] 10.222.0.33 () {56 vars in 1148 bytes} [Sun Oct 9 00:43:16 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:43:16.404677 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbd73dda-606d-427d-b082-97e9b95d2388 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:16.457286 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09ec3749-0547-4b36-bfb7-dca8eee5bde2 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:16.468648 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:16.494170 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0923e08-2bf9-4879-bfd0-dac122baebe3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:16.494892 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0923e08-2bf9-4879-bfd0-dac122baebe3 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:16.495247 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0923e08-2bf9-4879-bfd0-dac122baebe3 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:16.495614 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0923e08-2bf9-4879-bfd0-dac122baebe3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:16.506633 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0923e08-2bf9-4879-bfd0-dac122baebe3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:16.508785 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0923e08-2bf9-4879-bfd0-dac122baebe3 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:16.510754 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:16.526694 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-a0923e08-2bf9-4879-bfd0-dac122baebe3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:16.526694 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:16.526694 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:16.526694 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:16.526694 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:16.526694 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:16.526694 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:16.526694 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:16.526694 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:16.526694 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:16.526694 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:16.526694 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:16.526694 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:43:16.526694 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:43:16.526694 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:43:16.526694 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:43:16.527689 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:16.527689 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:16.527689 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:16.527689 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:16.527689 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:16.527689 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:16.527689 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:16.527689 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:16.527689 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:16.527689 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:16.527689 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:16.527689 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:16.527689 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:16.527689 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:16.527689 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:16.527689 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:16.527689 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:43:16.528572 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:43:16.528572 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:16.528572 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:16.528572 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 339/697] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:43:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:16.533758 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0923e08-2bf9-4879-bfd0-dac122baebe3 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:16.546549 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbd73dda-606d-427d-b082-97e9b95d2388 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:16.594788 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d87db30f-8456-41aa-93fa-80fcfba7b62a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:16.596211 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d87db30f-8456-41aa-93fa-80fcfba7b62a None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:16.596775 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d87db30f-8456-41aa-93fa-80fcfba7b62a None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:16.597253 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d87db30f-8456-41aa-93fa-80fcfba7b62a None admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:16.598687 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d87db30f-8456-41aa-93fa-80fcfba7b62a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:16.599176 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d87db30f-8456-41aa-93fa-80fcfba7b62a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:16.601263 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d87db30f-8456-41aa-93fa-80fcfba7b62a None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:16.618126 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 359/698] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:43:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:16.619852 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d87db30f-8456-41aa-93fa-80fcfba7b62a None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:16.644974 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0923e08-2bf9-4879-bfd0-dac122baebe3 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:16.683761 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47e4982a-edd2-4fe9-99a5-8831535b2692 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:16.684253 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e4982a-edd2-4fe9-99a5-8831535b2692 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:16.684601 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e4982a-edd2-4fe9-99a5-8831535b2692 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:16.684944 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e4982a-edd2-4fe9-99a5-8831535b2692 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:16.685723 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47e4982a-edd2-4fe9-99a5-8831535b2692 None admin] RBAC: Adding query filter params (role.id=228232e130c5412a80d774978175d28b, scope.project.id=013c83476b1340fbb051061833c4a9c6, user.id=70e367e802894b08afd7d756d8d9ecff) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:16.686100 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47e4982a-edd2-4fe9-99a5-8831535b2692 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:16.687585 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47e4982a-edd2-4fe9-99a5-8831535b2692 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:16.699427 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 340/699] 10.222.0.33 () {56 vars in 1374 bytes} [Sun Oct 9 00:43:16 2022] GET /identity/v3/role_assignments?user.id=70e367e802894b08afd7d756d8d9ecff&scope.project.id=013c83476b1340fbb051061833c4a9c6&role.id=228232e130c5412a80d774978175d28b => generated 608 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:16.699427 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47e4982a-edd2-4fe9-99a5-8831535b2692 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:16.792521 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:16.801628 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d87db30f-8456-41aa-93fa-80fcfba7b62a None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:16.807982 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:16.871127 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a91d94c-eebd-4d34-95f7-07eb6954e913 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:16.871675 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a91d94c-eebd-4d34-95f7-07eb6954e913 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:16.871955 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a91d94c-eebd-4d34-95f7-07eb6954e913 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:16.872222 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a91d94c-eebd-4d34-95f7-07eb6954e913 None admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:16.872861 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a91d94c-eebd-4d34-95f7-07eb6954e913 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:16.873157 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a91d94c-eebd-4d34-95f7-07eb6954e913 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:16.874550 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a91d94c-eebd-4d34-95f7-07eb6954e913 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:16.878555 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47e4982a-edd2-4fe9-99a5-8831535b2692 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:16.900836 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 360/700] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:43:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 281 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:16.908275 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a91d94c-eebd-4d34-95f7-07eb6954e913 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:16.948901 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7847ab5e-bd63-43bb-b1d1-b6ac70166fb8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:16.950470 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7847ab5e-bd63-43bb-b1d1-b6ac70166fb8 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:16.951226 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7847ab5e-bd63-43bb-b1d1-b6ac70166fb8 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:16.951767 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7847ab5e-bd63-43bb-b1d1-b6ac70166fb8 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:16.964383 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7847ab5e-bd63-43bb-b1d1-b6ac70166fb8 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:16.966205 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7847ab5e-bd63-43bb-b1d1-b6ac70166fb8 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:16.980826 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-7847ab5e-bd63-43bb-b1d1-b6ac70166fb8 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 09 00:43:16.980826 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:16.980826 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:16.980826 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:16.980826 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:16.980826 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:16.980826 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:16.980826 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:16.980826 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:16.980826 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:16.980826 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:16.980826 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:16.980826 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 09 00:43:16.980826 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 09 00:43:16.980826 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 09 00:43:16.980826 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 09 00:43:16.981784 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:16.981784 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:16.981784 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 09 00:43:16.981784 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 09 00:43:16.981784 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 09 00:43:16.981784 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 09 00:43:16.981784 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:16.981784 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:16.981784 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:16.981784 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:16.981784 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:16.981784 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:16.981784 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:16.981784 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:16.981784 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:16.981784 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:16.981784 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:16.981784 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:16.982736 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:16.982736 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:16.982736 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 09 00:43:16.982736 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 09 00:43:16.982736 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 09 00:43:16.982736 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 09 00:43:16.982736 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 09 00:43:16.982736 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 09 00:43:16.982736 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 09 00:43:16.982736 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:16.986299 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 341/701] 10.222.0.33 () {56 vars in 1093 bytes} [Sun Oct 9 00:43:16 2022] GET /identity/v3/users/nova => generated 82 bytes in 292 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:16.991473 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7847ab5e-bd63-43bb-b1d1-b6ac70166fb8 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:17.030825 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a91d94c-eebd-4d34-95f7-07eb6954e913 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:17.069196 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70b648b1-5868-4ce4-85ec-f59562a15273 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:17.070476 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70b648b1-5868-4ce4-85ec-f59562a15273 None admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:17.070753 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70b648b1-5868-4ce4-85ec-f59562a15273 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:17.071017 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70b648b1-5868-4ce4-85ec-f59562a15273 None admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:17.071700 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70b648b1-5868-4ce4-85ec-f59562a15273 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:17.073142 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70b648b1-5868-4ce4-85ec-f59562a15273 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:17.114771 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 361/702] 10.222.0.33 () {60 vars in 1132 bytes} [Sun Oct 9 00:43:16 2022] POST /identity/v3/users => generated 254 bytes in 207 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 09 00:43:17.123416 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7847ab5e-bd63-43bb-b1d1-b6ac70166fb8 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:17.146434 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:17.167487 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e047116-e413-4bd8-ac39-96a0c4471baa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:17.168153 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e047116-e413-4bd8-ac39-96a0c4471baa None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:17.168436 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e047116-e413-4bd8-ac39-96a0c4471baa None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:17.168710 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e047116-e413-4bd8-ac39-96a0c4471baa None admin] PATH_INFO: `/v3/users` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:17.169449 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e047116-e413-4bd8-ac39-96a0c4471baa None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:17.169756 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e047116-e413-4bd8-ac39-96a0c4471baa None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:17.171213 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e047116-e413-4bd8-ac39-96a0c4471baa None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:17.192511 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 342/703] 10.222.0.33 () {56 vars in 1133 bytes} [Sun Oct 9 00:43:16 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:43:17.198889 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70b648b1-5868-4ce4-85ec-f59562a15273 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:17.229903 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:17.329637 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70b648b1-5868-4ce4-85ec-f59562a15273 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:17.378396 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0791a853-a3d6-49fa-9d04-40621770fed7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:17.379344 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0791a853-a3d6-49fa-9d04-40621770fed7 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:17.379938 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0791a853-a3d6-49fa-9d04-40621770fed7 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:17.380396 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0791a853-a3d6-49fa-9d04-40621770fed7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:17.462337 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0791a853-a3d6-49fa-9d04-40621770fed7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:17.464271 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0791a853-a3d6-49fa-9d04-40621770fed7 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:17.486765 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:17.519384 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898be4a1-62fe-45c9-9830-28cf44e48bb6 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:17.519705 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898be4a1-62fe-45c9-9830-28cf44e48bb6 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:17.519985 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898be4a1-62fe-45c9-9830-28cf44e48bb6 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:17.521396 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 343/704] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:43:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 09 00:43:17.528833 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62c09802-d58d-427b-9c7c-1b0ec641e8da None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:17.529203 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62c09802-d58d-427b-9c7c-1b0ec641e8da None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:17.529552 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62c09802-d58d-427b-9c7c-1b0ec641e8da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:17.555325 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-62c09802-d58d-427b-9c7c-1b0ec641e8da None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:43:17.557033 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-62c09802-d58d-427b-9c7c-1b0ec641e8da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:43:17.585613 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 362/705] 10.222.0.33 () {58 vars in 1282 bytes} [Sun Oct 9 00:43:17 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 387 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 09 00:43:17.591586 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0791a853-a3d6-49fa-9d04-40621770fed7 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:17.637162 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 344/706] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:43:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 09 00:43:17.708295 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0791a853-a3d6-49fa-9d04-40621770fed7 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:17.732089 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:17.747016 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52734d48-286e-45a7-a8e0-6401a0c0567b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:17.747682 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52734d48-286e-45a7-a8e0-6401a0c0567b None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:17.747967 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52734d48-286e-45a7-a8e0-6401a0c0567b None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:17.748370 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52734d48-286e-45a7-a8e0-6401a0c0567b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:17.761842 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52734d48-286e-45a7-a8e0-6401a0c0567b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:17.763303 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52734d48-286e-45a7-a8e0-6401a0c0567b None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:17.779192 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-52734d48-286e-45a7-a8e0-6401a0c0567b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:17.779192 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:17.779192 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:17.779192 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:17.779192 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:17.779192 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:17.779192 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:17.779192 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:17.779192 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:17.779192 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:17.779192 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:17.779192 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:17.779192 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:43:17.779192 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:43:17.779192 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:43:17.779192 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:43:17.780192 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:17.780192 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:17.780192 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:17.780192 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:17.780192 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:17.780192 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:17.780192 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:17.780192 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:17.780192 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:17.780192 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:17.780192 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:17.780192 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:17.780192 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:17.780192 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:17.780192 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:17.780192 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:17.780192 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:43:17.781066 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:43:17.781066 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:17.781066 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:17.781066 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 363/707] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:43:17 2022] GET /identity/v3/domains/Default => generated 87 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 09 00:43:17.786247 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62c09802-d58d-427b-9c7c-1b0ec641e8da None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:17.895627 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62c09802-d58d-427b-9c7c-1b0ec641e8da None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:17.904710 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:17.937379 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50e8ee3d-6109-4caa-8a66-eb8d25a618b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:17.937993 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e8ee3d-6109-4caa-8a66-eb8d25a618b6 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:17.938275 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e8ee3d-6109-4caa-8a66-eb8d25a618b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:17.938552 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e8ee3d-6109-4caa-8a66-eb8d25a618b6 None admin] PATH_INFO: `/v3/domains` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:17.939199 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50e8ee3d-6109-4caa-8a66-eb8d25a618b6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:17.939502 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50e8ee3d-6109-4caa-8a66-eb8d25a618b6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:17.940973 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50e8ee3d-6109-4caa-8a66-eb8d25a618b6 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:17.953722 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 345/708] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:43:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:17.959269 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52734d48-286e-45a7-a8e0-6401a0c0567b None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:17.959846 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50e8ee3d-6109-4caa-8a66-eb8d25a618b6 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:18.109441 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50e8ee3d-6109-4caa-8a66-eb8d25a618b6 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:18.117481 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52734d48-286e-45a7-a8e0-6401a0c0567b None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:18.147958 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e094ef68-b8fc-4133-a89d-f10da3075b08 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:18.148509 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e094ef68-b8fc-4133-a89d-f10da3075b08 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:18.148786 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e094ef68-b8fc-4133-a89d-f10da3075b08 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:18.149303 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e094ef68-b8fc-4133-a89d-f10da3075b08 None admin] PATH_INFO: `/v3/services/network` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:18.149880 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e094ef68-b8fc-4133-a89d-f10da3075b08 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:18.152007 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e094ef68-b8fc-4133-a89d-f10da3075b08 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:18.162543 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a031b891-c491-49ef-af1c-090c2e6f3e4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:18.163782 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a031b891-c491-49ef-af1c-090c2e6f3e4e None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:18.164283 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a031b891-c491-49ef-af1c-090c2e6f3e4e None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:18.164776 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a031b891-c491-49ef-af1c-090c2e6f3e4e None admin] PATH_INFO: `/v3/projects/service` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:18.165579 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-e094ef68-b8fc-4133-a89d-f10da3075b08 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Oct 09 00:43:18.165579 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:18.165579 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 09 00:43:18.165579 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 09 00:43:18.165579 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 09 00:43:18.165579 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 09 00:43:18.165579 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 09 00:43:18.165579 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 09 00:43:18.165579 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 09 00:43:18.165579 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:18.165579 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 09 00:43:18.165579 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:18.165579 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:18.165579 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:18.165579 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:18.165579 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:18.165579 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:18.166546 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:18.166546 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:18.166546 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:18.166546 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:18.166546 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:18.166546 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:18.166546 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 09 00:43:18.166546 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 09 00:43:18.166546 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 09 00:43:18.166546 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 09 00:43:18.166546 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:18.166546 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:18.166546 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:18.166546 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:18.166546 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:18.166546 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:18.167387 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:18.167387 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:18.167387 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:18.167387 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:18.167387 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:18.167387 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:18.167387 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:18.167387 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:18.167387 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:18.167387 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:18.167387 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 09 00:43:18.167387 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 09 00:43:18.167387 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 09 00:43:18.167387 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:18.167387 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 346/709] 10.222.0.33 () {56 vars in 1111 bytes} [Sun Oct 9 00:43:17 2022] GET /identity/v3/services/network => generated 88 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 09 00:43:18.168763 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d19a609e-4e7e-4b31-a2f1-ab17a603b1dd None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:18.168763 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d19a609e-4e7e-4b31-a2f1-ab17a603b1dd None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:18.168932 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d19a609e-4e7e-4b31-a2f1-ab17a603b1dd None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:18.174501 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 347/710] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:43:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:43:18.175265 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d19a609e-4e7e-4b31-a2f1-ab17a603b1dd None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:18.177849 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a031b891-c491-49ef-af1c-090c2e6f3e4e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:18.181390 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a031b891-c491-49ef-af1c-090c2e6f3e4e None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:18.198224 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-a031b891-c491-49ef-af1c-090c2e6f3e4e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 09 00:43:18.198224 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:18.198224 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:18.198224 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:18.198224 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:18.198224 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:18.198224 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:18.198224 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:18.198224 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:18.198224 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:18.198224 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:18.198224 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:18.198224 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 09 00:43:18.198224 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 09 00:43:18.198224 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 09 00:43:18.199110 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 09 00:43:18.199110 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:18.199110 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:18.199110 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:18.199110 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:18.199110 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:18.199110 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:18.199110 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:18.199110 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:18.199110 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:18.199110 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:18.199110 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:18.199110 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:18.199110 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:18.199110 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:18.199110 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:18.199110 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:18.200036 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 09 00:43:18.200036 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 09 00:43:18.200036 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 09 00:43:18.200036 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 09 00:43:18.200036 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 09 00:43:18.200036 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 09 00:43:18.200036 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 09 00:43:18.200036 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:18.201178 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 364/711] 10.222.0.33 () {56 vars in 1111 bytes} [Sun Oct 9 00:43:17 2022] GET /identity/v3/projects/service => generated 88 bytes in 242 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:18.202491 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93e1b4ef-f720-47a1-bf06-a528301d5184 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:18.202902 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93e1b4ef-f720-47a1-bf06-a528301d5184 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:18.203209 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93e1b4ef-f720-47a1-bf06-a528301d5184 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:18.218443 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:18.222079 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-93e1b4ef-f720-47a1-bf06-a528301d5184 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:43:18.223335 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-93e1b4ef-f720-47a1-bf06-a528301d5184 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:43:18.255062 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:18.293669 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 365/712] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:43:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 09 00:43:18.294777 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93e1b4ef-f720-47a1-bf06-a528301d5184 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:18.308343 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d19a609e-4e7e-4b31-a2f1-ab17a603b1dd None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:18.376763 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c3dc41b-62b3-4611-9d8f-b1ba548b568d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:18.377377 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c3dc41b-62b3-4611-9d8f-b1ba548b568d None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:18.377653 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c3dc41b-62b3-4611-9d8f-b1ba548b568d None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:18.377923 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c3dc41b-62b3-4611-9d8f-b1ba548b568d None admin] PATH_INFO: `/v3/services` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:18.378541 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c3dc41b-62b3-4611-9d8f-b1ba548b568d None admin] RBAC: Adding query filter params (name=network) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:18.378838 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c3dc41b-62b3-4611-9d8f-b1ba548b568d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:18.380330 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c3dc41b-62b3-4611-9d8f-b1ba548b568d None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:18.390574 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 348/713] 10.222.0.33 () {56 vars in 1112 bytes} [Sun Oct 9 00:43:18 2022] GET /identity/v3/services?name=network => generated 124 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:18.398127 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c3dc41b-62b3-4611-9d8f-b1ba548b568d None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:18.424655 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93e1b4ef-f720-47a1-bf06-a528301d5184 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:18.468370 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f848d71c-4932-480e-9bfd-e926594cd650 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:18.469643 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f848d71c-4932-480e-9bfd-e926594cd650 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:18.469643 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f848d71c-4932-480e-9bfd-e926594cd650 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:18.470691 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f848d71c-4932-480e-9bfd-e926594cd650 None admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:18.471396 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f848d71c-4932-480e-9bfd-e926594cd650 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:18.471783 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f848d71c-4932-480e-9bfd-e926594cd650 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:18.473256 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f848d71c-4932-480e-9bfd-e926594cd650 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:18.487075 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 366/714] 10.222.0.33 () {56 vars in 1148 bytes} [Sun Oct 9 00:43:18 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:18.493052 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f848d71c-4932-480e-9bfd-e926594cd650 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:18.522623 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:18.602872 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f848d71c-4932-480e-9bfd-e926594cd650 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:18.627092 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:18.640831 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4041df50-468c-4fb3-84ee-2be800f1e8dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:18.641784 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4041df50-468c-4fb3-84ee-2be800f1e8dc None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:18.642341 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4041df50-468c-4fb3-84ee-2be800f1e8dc None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:18.642823 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4041df50-468c-4fb3-84ee-2be800f1e8dc None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:18.643790 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4041df50-468c-4fb3-84ee-2be800f1e8dc None admin] RBAC: Adding query filter params (role.id=092036ee46ac427baf4d56468575f697, scope.project.id=013c83476b1340fbb051061833c4a9c6, user.id=e0a5d95969e74157bb2d2a57207e1651) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:18.644302 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4041df50-468c-4fb3-84ee-2be800f1e8dc None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:18.646157 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4041df50-468c-4fb3-84ee-2be800f1e8dc None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:18.653135 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 367/715] 10.222.0.33 () {56 vars in 1374 bytes} [Sun Oct 9 00:43:18 2022] GET /identity/v3/role_assignments?user.id=e0a5d95969e74157bb2d2a57207e1651&scope.project.id=013c83476b1340fbb051061833c4a9c6&role.id=092036ee46ac427baf4d56468575f697 => generated 259 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:43:18.661935 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c3dc41b-62b3-4611-9d8f-b1ba548b568d None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:18.697846 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4041df50-468c-4fb3-84ee-2be800f1e8dc None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:18.733195 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa5c2d5f-e177-4195-8735-b4c114dc162f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:18.734505 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa5c2d5f-e177-4195-8735-b4c114dc162f None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:18.734690 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa5c2d5f-e177-4195-8735-b4c114dc162f None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:18.737486 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa5c2d5f-e177-4195-8735-b4c114dc162f None admin] PATH_INFO: `/v3/services` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:18.738185 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa5c2d5f-e177-4195-8735-b4c114dc162f None admin] RBAC: Adding query filter params (type=network) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:18.738486 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa5c2d5f-e177-4195-8735-b4c114dc162f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:18.740650 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa5c2d5f-e177-4195-8735-b4c114dc162f None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:18.754711 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 349/716] 10.222.0.33 () {56 vars in 1112 bytes} [Sun Oct 9 00:43:18 2022] GET /identity/v3/services?type=network => generated 124 bytes in 357 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:43:18.849161 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:18.871896 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4041df50-468c-4fb3-84ee-2be800f1e8dc None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:18.932469 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0ab19c4-df23-415e-bb4d-95ab3d7f4645 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:18.933109 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0ab19c4-df23-415e-bb4d-95ab3d7f4645 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:18.933383 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0ab19c4-df23-415e-bb4d-95ab3d7f4645 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:18.933650 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0ab19c4-df23-415e-bb4d-95ab3d7f4645 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:18.934230 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0ab19c4-df23-415e-bb4d-95ab3d7f4645 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:18.935818 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0ab19c4-df23-415e-bb4d-95ab3d7f4645 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:18.950749 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-c0ab19c4-df23-415e-bb4d-95ab3d7f4645 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 09 00:43:18.950749 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:18.950749 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 09 00:43:18.950749 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 09 00:43:18.950749 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 09 00:43:18.950749 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 09 00:43:18.950749 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 09 00:43:18.950749 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 09 00:43:18.950749 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 09 00:43:18.950749 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:18.950749 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 09 00:43:18.950749 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:18.950749 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:18.950749 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:18.950749 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:18.950749 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:18.950749 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:18.951902 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:18.951902 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:18.951902 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:18.951902 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:18.951902 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:18.951902 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:18.951902 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 09 00:43:18.951902 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 09 00:43:18.951902 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 09 00:43:18.951902 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 09 00:43:18.951902 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:18.951902 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:18.951902 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:18.951902 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:18.951902 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:18.951902 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:18.952761 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:18.952761 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:18.952761 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:18.952761 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:18.952761 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:18.952761 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:18.952761 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:18.952761 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:18.952761 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:18.952761 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:18.952761 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 09 00:43:18.952761 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 09 00:43:18.952761 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 09 00:43:18.952761 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:18.955734 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 368/717] 10.222.0.33 () {56 vars in 1129 bytes} [Sun Oct 9 00:43:18 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 255 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:18.958339 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa5c2d5f-e177-4195-8735-b4c114dc162f None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:19.070065 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa5c2d5f-e177-4195-8735-b4c114dc162f None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:19.126647 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-632e38b6-f2f9-4130-8dff-0c1aa17eeae3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:19.127265 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-632e38b6-f2f9-4130-8dff-0c1aa17eeae3 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:19.127546 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-632e38b6-f2f9-4130-8dff-0c1aa17eeae3 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:19.127851 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-632e38b6-f2f9-4130-8dff-0c1aa17eeae3 None admin] PATH_INFO: `/v3/services` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:19.128481 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-632e38b6-f2f9-4130-8dff-0c1aa17eeae3 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:19.128785 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-632e38b6-f2f9-4130-8dff-0c1aa17eeae3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:19.130155 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-632e38b6-f2f9-4130-8dff-0c1aa17eeae3 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:19.136954 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 350/718] 10.222.0.33 () {56 vars in 1124 bytes} [Sun Oct 9 00:43:18 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:19.142470 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0ab19c4-df23-415e-bb4d-95ab3d7f4645 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:19.290596 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0ab19c4-df23-415e-bb4d-95ab3d7f4645 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:19.338149 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f11eda6-46b4-468b-92f1-716bc70bb943 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:19.338754 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f11eda6-46b4-468b-92f1-716bc70bb943 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:19.339031 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f11eda6-46b4-468b-92f1-716bc70bb943 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:19.339296 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f11eda6-46b4-468b-92f1-716bc70bb943 None admin] PATH_INFO: `/v3/services` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:19.339968 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f11eda6-46b4-468b-92f1-716bc70bb943 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:19.340256 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f11eda6-46b4-468b-92f1-716bc70bb943 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:19.341770 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f11eda6-46b4-468b-92f1-716bc70bb943 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:19.349117 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 369/719] 10.222.0.33 () {56 vars in 1124 bytes} [Sun Oct 9 00:43:19 2022] GET /identity/v3/services?type=block-storage => generated 130 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:19.508912 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c6adc2f-b2e7-4e6c-adda-e65f19510b5f None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:19.509272 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c6adc2f-b2e7-4e6c-adda-e65f19510b5f None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:19.509476 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c6adc2f-b2e7-4e6c-adda-e65f19510b5f None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:19.511084 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 351/720] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:43:19 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:43:19.516831 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6228b6c7-cd70-451e-b3f2-d5cbf3dd5c94 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:19.517343 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6228b6c7-cd70-451e-b3f2-d5cbf3dd5c94 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:19.517343 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6228b6c7-cd70-451e-b3f2-d5cbf3dd5c94 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:19.543269 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-6228b6c7-cd70-451e-b3f2-d5cbf3dd5c94 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:43:19.544429 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-6228b6c7-cd70-451e-b3f2-d5cbf3dd5c94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:43:19.557670 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:19.634969 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 370/721] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:43:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 09 00:43:20.125269 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c6adc2f-b2e7-4e6c-adda-e65f19510b5f None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:20.158846 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:20.239554 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c6adc2f-b2e7-4e6c-adda-e65f19510b5f None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:20.275172 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c55285-6a4b-4fea-b5a2-0f09c0ba5500 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:20.275833 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c55285-6a4b-4fea-b5a2-0f09c0ba5500 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:20.276178 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c55285-6a4b-4fea-b5a2-0f09c0ba5500 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:20.276451 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c55285-6a4b-4fea-b5a2-0f09c0ba5500 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:20.287374 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17c55285-6a4b-4fea-b5a2-0f09c0ba5500 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:43:20.293761 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-17c55285-6a4b-4fea-b5a2-0f09c0ba5500 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 09 00:43:20.293761 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:20.293761 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:20.293761 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:20.293761 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:20.293761 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:20.293761 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:20.293761 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:20.293761 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:20.293761 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:20.293761 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:20.293761 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:20.293761 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:43:20.293761 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:43:20.293761 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:43:20.293761 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:43:20.294690 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:43:20.294690 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:43:20.294690 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:43:20.294690 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:43:20.294690 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:20.294690 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:20.294690 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:20.294690 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:20.294690 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:20.294690 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:20.294690 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:20.294690 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:20.294690 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:20.294690 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:20.294690 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:20.294690 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:20.295815 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:20.295815 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:20.295815 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:20.295815 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:20.295815 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:43:20.295815 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:43:20.295815 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:43:20.295815 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:43:20.295815 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:43:20.295815 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:43:20.295815 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 09 00:43:20.295815 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:20.295815 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 352/722] 10.222.0.33 () {56 vars in 1096 bytes} [Sun Oct 9 00:43:20 2022] GET /identity/v3/roles/admin => generated 83 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:20.302605 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6228b6c7-cd70-451e-b3f2-d5cbf3dd5c94 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:20.321018 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93912bf2-3a5f-4c25-9a67-65ab7f041069 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:20.321359 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93912bf2-3a5f-4c25-9a67-65ab7f041069 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:20.321643 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93912bf2-3a5f-4c25-9a67-65ab7f041069 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:20.322971 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 353/723] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:43:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 09 00:43:20.327839 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907da8d6-014c-475b-898e-50a0e47cb009 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:20.328121 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907da8d6-014c-475b-898e-50a0e47cb009 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:20.328395 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907da8d6-014c-475b-898e-50a0e47cb009 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:20.359806 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-907da8d6-014c-475b-898e-50a0e47cb009 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:43:20.361307 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-907da8d6-014c-475b-898e-50a0e47cb009 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:43:20.447667 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6228b6c7-cd70-451e-b3f2-d5cbf3dd5c94 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:20.449870 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 354/724] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:43:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 09 00:43:20.484905 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecad5fa7-6731-4a4e-98da-5bfdb8903dfd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:20.485749 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecad5fa7-6731-4a4e-98da-5bfdb8903dfd None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:20.486167 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecad5fa7-6731-4a4e-98da-5bfdb8903dfd None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:20.486613 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecad5fa7-6731-4a4e-98da-5bfdb8903dfd None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:20.487407 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecad5fa7-6731-4a4e-98da-5bfdb8903dfd None admin] RBAC: Adding query filter params (name=admin) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:20.487878 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecad5fa7-6731-4a4e-98da-5bfdb8903dfd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:20.489495 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecad5fa7-6731-4a4e-98da-5bfdb8903dfd None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:20.502927 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 371/725] 10.222.0.33 () {56 vars in 1099 bytes} [Sun Oct 9 00:43:20 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:20.504610 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907da8d6-014c-475b-898e-50a0e47cb009 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:20.527441 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:20.621963 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907da8d6-014c-475b-898e-50a0e47cb009 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:20.668814 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8838e512-f29e-410b-92ac-083623851e1a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:20.669461 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8838e512-f29e-410b-92ac-083623851e1a None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:20.669747 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8838e512-f29e-410b-92ac-083623851e1a None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:20.670027 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8838e512-f29e-410b-92ac-083623851e1a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:20.684814 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8838e512-f29e-410b-92ac-083623851e1a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:20.686287 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8838e512-f29e-410b-92ac-083623851e1a None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:20.716688 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-8838e512-f29e-410b-92ac-083623851e1a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:20.716688 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:20.716688 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:20.716688 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:20.716688 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:20.716688 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:20.716688 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:20.716688 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:20.716688 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:20.716688 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:20.716688 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:20.716688 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:20.716688 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:43:20.716688 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:43:20.716688 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:43:20.716688 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:43:20.717655 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:20.717655 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:20.717655 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:20.717655 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:20.717655 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:20.717655 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:20.717655 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:20.717655 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:20.717655 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:20.717655 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:20.717655 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:20.717655 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:20.717655 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:20.717655 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:20.717655 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:20.717655 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:20.717655 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:43:20.719137 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:43:20.719137 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:20.719137 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:20.719137 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 355/726] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:43:20 2022] GET /identity/v3/domains/Default => generated 87 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:20.725839 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecad5fa7-6731-4a4e-98da-5bfdb8903dfd None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:20.751475 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:20.800471 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8838e512-f29e-410b-92ac-083623851e1a None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:20.923054 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:20.958318 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecad5fa7-6731-4a4e-98da-5bfdb8903dfd None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:21.017585 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8838e512-f29e-410b-92ac-083623851e1a None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:21.034216 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aa47948-d1e7-4f22-959d-8f7e425fe402 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:21.034826 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa47948-d1e7-4f22-959d-8f7e425fe402 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:21.035100 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa47948-d1e7-4f22-959d-8f7e425fe402 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:21.035367 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa47948-d1e7-4f22-959d-8f7e425fe402 None admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:21.036027 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aa47948-d1e7-4f22-959d-8f7e425fe402 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:21.036319 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aa47948-d1e7-4f22-959d-8f7e425fe402 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:21.037682 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aa47948-d1e7-4f22-959d-8f7e425fe402 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:21.063119 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 372/727] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:43:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 337 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:21.066568 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aa47948-d1e7-4f22-959d-8f7e425fe402 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:21.094975 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cc78d55-36e3-4c05-a403-c0b8fdf0199e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:21.095624 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc78d55-36e3-4c05-a403-c0b8fdf0199e None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:21.095916 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc78d55-36e3-4c05-a403-c0b8fdf0199e None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:21.096188 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc78d55-36e3-4c05-a403-c0b8fdf0199e None admin] PATH_INFO: `/v3/roles/service` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:21.110284 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cc78d55-36e3-4c05-a403-c0b8fdf0199e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:43:21.118221 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-5cc78d55-36e3-4c05-a403-c0b8fdf0199e None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 09 00:43:21.118221 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:21.118221 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:21.118221 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:21.118221 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:21.118221 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:21.118221 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:21.118221 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:21.118221 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:21.118221 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:21.118221 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:21.118221 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:21.118221 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:43:21.118221 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:43:21.118221 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:43:21.118221 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:43:21.119179 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:43:21.119179 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:43:21.119179 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:43:21.119179 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:43:21.119179 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:21.119179 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:21.119179 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:21.119179 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:21.119179 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:21.119179 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:21.119179 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:21.119179 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:21.119179 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:21.119179 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:21.119179 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:21.119179 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:21.120054 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:21.120054 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:21.120054 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:21.120054 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:21.120054 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:43:21.120054 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:43:21.120054 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:43:21.120054 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:43:21.120054 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:43:21.120054 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:43:21.120054 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 09 00:43:21.120054 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:21.125324 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 356/728] 10.222.0.33 () {56 vars in 1102 bytes} [Sun Oct 9 00:43:20 2022] GET /identity/v3/roles/service => generated 85 bytes in 320 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 09 00:43:21.127106 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cc78d55-36e3-4c05-a403-c0b8fdf0199e None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:21.289918 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:21.336640 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cc78d55-36e3-4c05-a403-c0b8fdf0199e None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:21.363969 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:21.381875 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aa47948-d1e7-4f22-959d-8f7e425fe402 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:21.386173 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3edcf76b-4fb3-456a-82d7-63bdca363de1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:21.387085 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3edcf76b-4fb3-456a-82d7-63bdca363de1 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:21.387576 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3edcf76b-4fb3-456a-82d7-63bdca363de1 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:21.388839 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3edcf76b-4fb3-456a-82d7-63bdca363de1 None admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:21.389706 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3edcf76b-4fb3-456a-82d7-63bdca363de1 None admin] RBAC: Adding query filter params (name=service) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:21.390746 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3edcf76b-4fb3-456a-82d7-63bdca363de1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:21.396608 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3edcf76b-4fb3-456a-82d7-63bdca363de1 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:21.408788 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 357/729] 10.222.0.33 () {56 vars in 1103 bytes} [Sun Oct 9 00:43:21 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 282 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:43:21.414248 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822d5642-da6a-435a-a22a-4d381ce5468b None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:21.414248 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822d5642-da6a-435a-a22a-4d381ce5468b None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:21.414248 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822d5642-da6a-435a-a22a-4d381ce5468b None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:21.414248 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 358/730] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:43:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:43:21.421306 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-822d5642-da6a-435a-a22a-4d381ce5468b None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:21.441766 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acb20121-7289-436b-bfba-95e9fc40a828 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:21.443070 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb20121-7289-436b-bfba-95e9fc40a828 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:21.443822 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb20121-7289-436b-bfba-95e9fc40a828 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:21.444378 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb20121-7289-436b-bfba-95e9fc40a828 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:21.453982 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acb20121-7289-436b-bfba-95e9fc40a828 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:21.456136 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acb20121-7289-436b-bfba-95e9fc40a828 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:21.472039 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-acb20121-7289-436b-bfba-95e9fc40a828 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 09 00:43:21.472039 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:21.472039 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:21.472039 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:21.472039 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:21.472039 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:21.472039 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:21.472039 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:21.472039 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:21.472039 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:21.472039 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:21.472039 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:21.472039 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 09 00:43:21.472039 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 09 00:43:21.472039 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 09 00:43:21.472039 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 09 00:43:21.473043 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:21.473043 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:21.473043 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 09 00:43:21.473043 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 09 00:43:21.473043 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 09 00:43:21.473043 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 09 00:43:21.473043 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:21.473043 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:21.473043 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:21.473043 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:21.473043 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:21.473043 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:21.473043 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:21.473043 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:21.473043 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:21.473043 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:21.473043 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:21.473043 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:21.473977 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:21.473977 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:21.473977 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 09 00:43:21.473977 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 09 00:43:21.473977 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 09 00:43:21.473977 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 09 00:43:21.473977 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 09 00:43:21.473977 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 09 00:43:21.473977 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 09 00:43:21.473977 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:21.477468 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 373/731] 10.222.0.33 () {56 vars in 1096 bytes} [Sun Oct 9 00:43:21 2022] GET /identity/v3/users/swift => generated 83 bytes in 411 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:21.478982 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cee1e6d-80ab-455c-aa52-f1a0b324a905 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:21.479408 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cee1e6d-80ab-455c-aa52-f1a0b324a905 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:21.479846 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cee1e6d-80ab-455c-aa52-f1a0b324a905 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:21.504655 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-8cee1e6d-80ab-455c-aa52-f1a0b324a905 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:43:21.506490 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-8cee1e6d-80ab-455c-aa52-f1a0b324a905 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:43:21.562317 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-822d5642-da6a-435a-a22a-4d381ce5468b None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:21.594626 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 374/732] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:43:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 09 00:43:21.595727 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cee1e6d-80ab-455c-aa52-f1a0b324a905 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:21.612821 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b451e9e-6761-434b-af49-c34b764f91c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:21.613716 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b451e9e-6761-434b-af49-c34b764f91c8 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:21.613999 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b451e9e-6761-434b-af49-c34b764f91c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:21.614274 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b451e9e-6761-434b-af49-c34b764f91c8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:21.629649 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b451e9e-6761-434b-af49-c34b764f91c8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:21.631889 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b451e9e-6761-434b-af49-c34b764f91c8 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:21.650865 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-5b451e9e-6761-434b-af49-c34b764f91c8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:21.650865 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:21.650865 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:21.650865 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:21.650865 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:21.650865 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:21.650865 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:21.650865 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:21.650865 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:21.650865 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:21.650865 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:21.650865 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:21.650865 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:43:21.650865 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:43:21.650865 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:43:21.650865 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:43:21.651891 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:21.651891 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:21.651891 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:21.651891 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:21.651891 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:21.651891 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:21.651891 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:21.651891 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:21.651891 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:21.651891 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:21.651891 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:21.651891 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:21.651891 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:21.651891 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:21.651891 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:21.651891 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:21.651891 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:43:21.653003 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:43:21.653003 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:21.653003 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:21.653003 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 359/733] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:43:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 232 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:21.657585 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da7cd1e-a9c5-46ef-a877-9d670f65854e None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:21.657585 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da7cd1e-a9c5-46ef-a877-9d670f65854e None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:21.657585 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da7cd1e-a9c5-46ef-a877-9d670f65854e None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:21.657585 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 360/734] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:43:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:43:21.658592 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3da7cd1e-a9c5-46ef-a877-9d670f65854e None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:21.673771 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:21.709951 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:21.719209 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cee1e6d-80ab-455c-aa52-f1a0b324a905 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:21.755449 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdfbd1b1-ac13-4582-ac42-4683fc92cc32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:21.756106 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdfbd1b1-ac13-4582-ac42-4683fc92cc32 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:21.756353 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdfbd1b1-ac13-4582-ac42-4683fc92cc32 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:21.756626 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdfbd1b1-ac13-4582-ac42-4683fc92cc32 None admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:21.757434 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdfbd1b1-ac13-4582-ac42-4683fc92cc32 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:21.757740 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdfbd1b1-ac13-4582-ac42-4683fc92cc32 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:21.759177 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdfbd1b1-ac13-4582-ac42-4683fc92cc32 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:21.782774 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 375/735] 10.222.0.33 () {56 vars in 1135 bytes} [Sun Oct 9 00:43:21 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:21.783669 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3da7cd1e-a9c5-46ef-a877-9d670f65854e None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:21.783957 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5198360-6c93-47da-86c9-29c7b29904c4 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:21.784246 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5198360-6c93-47da-86c9-29c7b29904c4 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:21.784516 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5198360-6c93-47da-86c9-29c7b29904c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:21.802603 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-b5198360-6c93-47da-86c9-29c7b29904c4 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:43:21.804325 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-b5198360-6c93-47da-86c9-29c7b29904c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:43:21.823212 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad94ee96-757f-4a50-b0b6-8bc3eb182a3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:21.824171 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad94ee96-757f-4a50-b0b6-8bc3eb182a3a None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:21.824662 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad94ee96-757f-4a50-b0b6-8bc3eb182a3a None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:21.825143 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad94ee96-757f-4a50-b0b6-8bc3eb182a3a None admin] PATH_INFO: `/v3/domains` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:21.826092 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad94ee96-757f-4a50-b0b6-8bc3eb182a3a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:21.826624 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad94ee96-757f-4a50-b0b6-8bc3eb182a3a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:21.828217 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad94ee96-757f-4a50-b0b6-8bc3eb182a3a None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:21.846248 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 361/736] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:43:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:21.847422 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad94ee96-757f-4a50-b0b6-8bc3eb182a3a None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:21.875921 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 376/737] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:43:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 09 00:43:21.876760 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5198360-6c93-47da-86c9-29c7b29904c4 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:21.926416 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:21.952447 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:21.955999 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad94ee96-757f-4a50-b0b6-8bc3eb182a3a None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:21.993200 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afab10c4-8d11-4800-8873-1a1afcf24b32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:21.993996 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afab10c4-8d11-4800-8873-1a1afcf24b32 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:21.994372 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afab10c4-8d11-4800-8873-1a1afcf24b32 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:21.994705 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afab10c4-8d11-4800-8873-1a1afcf24b32 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:21.997650 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5198360-6c93-47da-86c9-29c7b29904c4 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:22.051040 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fc2c4af-b586-4055-9119-4261254d4f57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:22.051826 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc2c4af-b586-4055-9119-4261254d4f57 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:22.052384 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc2c4af-b586-4055-9119-4261254d4f57 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:22.052784 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc2c4af-b586-4055-9119-4261254d4f57 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:22.062067 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fc2c4af-b586-4055-9119-4261254d4f57 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:22.063524 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fc2c4af-b586-4055-9119-4261254d4f57 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:22.079329 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afab10c4-8d11-4800-8873-1a1afcf24b32 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:22.080759 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afab10c4-8d11-4800-8873-1a1afcf24b32 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:22.084545 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-1fc2c4af-b586-4055-9119-4261254d4f57 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 09 00:43:22.084545 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:22.084545 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:22.084545 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:22.084545 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:22.084545 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:22.084545 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:22.084545 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:22.084545 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:22.084545 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:22.084545 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:22.084545 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:22.084545 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 09 00:43:22.084545 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 09 00:43:22.084545 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 09 00:43:22.084545 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 09 00:43:22.085588 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:22.085588 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:22.085588 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 09 00:43:22.085588 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 09 00:43:22.085588 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 09 00:43:22.085588 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 09 00:43:22.085588 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:22.085588 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:22.085588 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:22.085588 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:22.085588 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:22.085588 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:22.085588 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:22.085588 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:22.085588 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:22.085588 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:22.085588 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:22.085588 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:22.086506 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:22.086506 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:22.086506 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 09 00:43:22.086506 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 09 00:43:22.086506 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 09 00:43:22.086506 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 09 00:43:22.086506 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 09 00:43:22.086506 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 09 00:43:22.086506 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 09 00:43:22.086506 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:22.089404 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 377/738] 10.222.0.33 () {56 vars in 1117 bytes} [Sun Oct 9 00:43:21 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:22.090470 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fc2c4af-b586-4055-9119-4261254d4f57 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:22.190803 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:22.196177 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 362/739] 10.222.0.33 () {58 vars in 1282 bytes} [Sun Oct 9 00:43:21 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 349 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 09 00:43:22.196976 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afab10c4-8d11-4800-8873-1a1afcf24b32 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:22.198455 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fc2c4af-b586-4055-9119-4261254d4f57 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:22.201497 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:22.232817 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49d969c7-3a62-4440-a44e-ae38db33f0aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:22.233437 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d969c7-3a62-4440-a44e-ae38db33f0aa None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:22.233722 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d969c7-3a62-4440-a44e-ae38db33f0aa None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:22.233996 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d969c7-3a62-4440-a44e-ae38db33f0aa None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:22.244484 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49d969c7-3a62-4440-a44e-ae38db33f0aa None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:43:22.249632 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-49d969c7-3a62-4440-a44e-ae38db33f0aa None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 09 00:43:22.249632 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:22.249632 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:22.249632 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:22.249632 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:22.249632 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:22.249632 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:22.249632 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:22.249632 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:22.249632 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:22.249632 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:22.249632 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:22.249632 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:43:22.249632 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:43:22.249632 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:43:22.249632 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:43:22.250545 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:43:22.250545 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:43:22.250545 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:43:22.250545 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:43:22.250545 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:22.250545 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:22.250545 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:22.250545 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:22.250545 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:22.250545 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:22.250545 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:22.250545 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:22.250545 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:22.250545 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:22.250545 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:22.250545 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:22.256699 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:22.256699 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:22.256699 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:22.256699 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:22.256699 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:43:22.256699 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:43:22.256699 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:43:22.256699 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:43:22.256699 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:43:22.256699 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:43:22.256699 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 09 00:43:22.256699 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:22.256699 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 378/740] 10.222.0.33 () {56 vars in 1096 bytes} [Sun Oct 9 00:43:22 2022] GET /identity/v3/roles/admin => generated 83 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:22.256699 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b45fc7-eeab-4e5d-a67b-4fa3b87812fc None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:22.257895 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b45fc7-eeab-4e5d-a67b-4fa3b87812fc None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:22.257895 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b45fc7-eeab-4e5d-a67b-4fa3b87812fc None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:22.257895 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 379/741] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:43:22 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:43:22.257895 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5b45fc7-eeab-4e5d-a67b-4fa3b87812fc None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:22.299660 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afab10c4-8d11-4800-8873-1a1afcf24b32 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:22.332952 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28674f8f-3d4f-4e19-b4fd-c1df74aae81a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:22.333489 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28674f8f-3d4f-4e19-b4fd-c1df74aae81a None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:22.333769 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28674f8f-3d4f-4e19-b4fd-c1df74aae81a None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:22.334043 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28674f8f-3d4f-4e19-b4fd-c1df74aae81a None admin] PATH_INFO: `/v3/users` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:22.334771 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28674f8f-3d4f-4e19-b4fd-c1df74aae81a None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:22.335101 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28674f8f-3d4f-4e19-b4fd-c1df74aae81a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:22.336551 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28674f8f-3d4f-4e19-b4fd-c1df74aae81a None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:22.353532 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 363/742] 10.222.0.33 () {56 vars in 1149 bytes} [Sun Oct 9 00:43:22 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:22.354482 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28674f8f-3d4f-4e19-b4fd-c1df74aae81a None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:22.354623 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5b45fc7-eeab-4e5d-a67b-4fa3b87812fc None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:22.387432 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccf83b06-53b2-4ddf-bd3a-a984964e3ff0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:22.388067 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccf83b06-53b2-4ddf-bd3a-a984964e3ff0 None admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:22.388346 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccf83b06-53b2-4ddf-bd3a-a984964e3ff0 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:22.388617 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccf83b06-53b2-4ddf-bd3a-a984964e3ff0 None admin] PATH_INFO: `/v3/services` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:22.389254 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccf83b06-53b2-4ddf-bd3a-a984964e3ff0 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:22.390603 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccf83b06-53b2-4ddf-bd3a-a984964e3ff0 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:22.405922 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 380/743] 10.222.0.33 () {60 vars in 1141 bytes} [Sun Oct 9 00:43:22 2022] POST /identity/v3/services => generated 240 bytes in 152 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:43:22.409897 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:22.414412 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccf83b06-53b2-4ddf-bd3a-a984964e3ff0 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:22.450348 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:22.458973 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28674f8f-3d4f-4e19-b4fd-c1df74aae81a None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:22.510896 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a02a317a-aac5-4c21-a79b-514343e21f3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:22.511725 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a02a317a-aac5-4c21-a79b-514343e21f3c None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:22.512015 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a02a317a-aac5-4c21-a79b-514343e21f3c None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:22.512280 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a02a317a-aac5-4c21-a79b-514343e21f3c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:22.525151 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a02a317a-aac5-4c21-a79b-514343e21f3c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:22.526772 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a02a317a-aac5-4c21-a79b-514343e21f3c None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:22.539352 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccf83b06-53b2-4ddf-bd3a-a984964e3ff0 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:22.543933 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-a02a317a-aac5-4c21-a79b-514343e21f3c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:22.543933 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:22.543933 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:22.543933 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:22.543933 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:22.543933 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:22.543933 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:22.543933 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:22.543933 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:22.543933 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:22.543933 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:22.543933 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:22.543933 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:43:22.543933 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:43:22.543933 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:43:22.543933 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:43:22.544871 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:22.544871 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:22.544871 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:22.544871 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:22.544871 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:22.544871 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:22.544871 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:22.544871 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:22.544871 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:22.544871 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:22.544871 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:22.544871 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:22.544871 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:22.544871 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:22.544871 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:22.544871 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:22.544871 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:43:22.551068 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:43:22.551068 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:22.551068 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:22.551068 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 364/744] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:43:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:22.551068 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e5f7c7-d3d6-495a-88a4-9ed8d821df56 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:22.551068 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e5f7c7-d3d6-495a-88a4-9ed8d821df56 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:22.551068 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e5f7c7-d3d6-495a-88a4-9ed8d821df56 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:22.568011 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-92e5f7c7-d3d6-495a-88a4-9ed8d821df56 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:43:22.569267 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-92e5f7c7-d3d6-495a-88a4-9ed8d821df56 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:43:22.583155 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5faf3b37-ba45-41e7-9834-ec989fe04380 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:22.583732 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5faf3b37-ba45-41e7-9834-ec989fe04380 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:22.584013 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5faf3b37-ba45-41e7-9834-ec989fe04380 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:22.584286 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5faf3b37-ba45-41e7-9834-ec989fe04380 None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:22.584953 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5faf3b37-ba45-41e7-9834-ec989fe04380 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:22.585789 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5faf3b37-ba45-41e7-9834-ec989fe04380 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:22.587165 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5faf3b37-ba45-41e7-9834-ec989fe04380 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:22.597242 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 381/745] 10.222.0.33 () {56 vars in 1099 bytes} [Sun Oct 9 00:43:22 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:22.597828 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5faf3b37-ba45-41e7-9834-ec989fe04380 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:22.648069 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 365/746] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:43:22 2022] POST /identity/v3/auth/tokens => generated 898 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 09 00:43:22.649317 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92e5f7c7-d3d6-495a-88a4-9ed8d821df56 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:22.684585 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:22.711525 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:22.733480 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5faf3b37-ba45-41e7-9834-ec989fe04380 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:22.757556 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92e5f7c7-d3d6-495a-88a4-9ed8d821df56 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:22.781596 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d08fc81f-9a70-4fb0-a5a4-70447103e583 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:22.783561 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d08fc81f-9a70-4fb0-a5a4-70447103e583 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:22.784034 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d08fc81f-9a70-4fb0-a5a4-70447103e583 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:22.784464 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d08fc81f-9a70-4fb0-a5a4-70447103e583 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:22.794487 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7b7964f-b4bd-4de0-bc82-c59988cd3fc4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:22.795237 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b7964f-b4bd-4de0-bc82-c59988cd3fc4 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:22.795633 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b7964f-b4bd-4de0-bc82-c59988cd3fc4 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:22.796040 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b7964f-b4bd-4de0-bc82-c59988cd3fc4 None admin] PATH_INFO: `/v3/domains` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:22.796877 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7b7964f-b4bd-4de0-bc82-c59988cd3fc4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:22.797269 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7b7964f-b4bd-4de0-bc82-c59988cd3fc4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:22.798712 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7b7964f-b4bd-4de0-bc82-c59988cd3fc4 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:22.811842 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 366/747] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:43:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:22.812809 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7b7964f-b4bd-4de0-bc82-c59988cd3fc4 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:22.881790 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d08fc81f-9a70-4fb0-a5a4-70447103e583 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:22.884125 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d08fc81f-9a70-4fb0-a5a4-70447103e583 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:22.929593 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:22.936692 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7b7964f-b4bd-4de0-bc82-c59988cd3fc4 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:22.972541 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36de0837-aa0f-4eb2-9e84-0ed547a3a8b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:22.973170 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36de0837-aa0f-4eb2-9e84-0ed547a3a8b2 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:22.973456 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36de0837-aa0f-4eb2-9e84-0ed547a3a8b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:22.973730 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36de0837-aa0f-4eb2-9e84-0ed547a3a8b2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:22.987550 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36de0837-aa0f-4eb2-9e84-0ed547a3a8b2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:22.989045 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36de0837-aa0f-4eb2-9e84-0ed547a3a8b2 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:22.992664 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:23.004565 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-36de0837-aa0f-4eb2-9e84-0ed547a3a8b2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:23.004565 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:23.004565 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:23.004565 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:23.004565 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:23.004565 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:23.004565 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:23.004565 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:23.004565 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:23.004565 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:23.004565 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:23.004565 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:23.004565 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:43:23.004565 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:43:23.004565 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:43:23.004565 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:43:23.005507 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:23.005507 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:23.005507 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:23.005507 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:23.005507 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:23.005507 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:23.005507 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:23.005507 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:23.005507 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:23.005507 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:23.005507 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:23.005507 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:23.005507 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:23.005507 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:23.005507 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:23.005507 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:23.005507 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:43:23.006366 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:43:23.006366 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:23.006366 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:23.006366 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 367/748] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:43:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:23.007200 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36de0837-aa0f-4eb2-9e84-0ed547a3a8b2 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:23.020274 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 382/749] 10.222.0.33 () {58 vars in 1282 bytes} [Sun Oct 9 00:43:22 2022] GET /identity/v3/auth/tokens => generated 898 bytes in 419 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 09 00:43:23.020274 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d08fc81f-9a70-4fb0-a5a4-70447103e583 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:23.121171 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d08fc81f-9a70-4fb0-a5a4-70447103e583 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:23.125426 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36de0837-aa0f-4eb2-9e84-0ed547a3a8b2 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:23.157227 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c94a635-1c45-4dd0-a252-f8f725e3be50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:23.157910 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c94a635-1c45-4dd0-a252-f8f725e3be50 None admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:23.158191 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c94a635-1c45-4dd0-a252-f8f725e3be50 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:23.158778 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c94a635-1c45-4dd0-a252-f8f725e3be50 None admin] PATH_INFO: `/v3/services` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:23.159427 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c94a635-1c45-4dd0-a252-f8f725e3be50 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:23.160830 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c94a635-1c45-4dd0-a252-f8f725e3be50 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:23.163400 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ab82704-92b5-4e95-b279-99b9902d9ce8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:23.164062 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab82704-92b5-4e95-b279-99b9902d9ce8 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:23.164470 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab82704-92b5-4e95-b279-99b9902d9ce8 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:23.164744 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab82704-92b5-4e95-b279-99b9902d9ce8 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:23.172117 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ab82704-92b5-4e95-b279-99b9902d9ce8 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:23.173461 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ab82704-92b5-4e95-b279-99b9902d9ce8 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:23.175739 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 383/750] 10.222.0.33 () {60 vars in 1141 bytes} [Sun Oct 9 00:43:23 2022] POST /identity/v3/services => generated 251 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:43:23.185453 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-6ab82704-92b5-4e95-b279-99b9902d9ce8 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 09 00:43:23.185453 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:23.185453 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:23.185453 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:23.185453 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:23.185453 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:23.185453 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:23.185453 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:23.185453 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:23.185453 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:23.185453 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:23.185453 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:23.185453 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 09 00:43:23.185453 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 09 00:43:23.185453 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 09 00:43:23.186419 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 09 00:43:23.186419 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:23.186419 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:23.186419 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:23.186419 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:23.186419 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:23.186419 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:23.186419 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:23.186419 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:23.186419 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:23.186419 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:23.186419 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:23.186419 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:23.186419 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:23.186419 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:23.186419 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:23.186419 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:23.187296 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 09 00:43:23.187296 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 09 00:43:23.187296 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 09 00:43:23.187296 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 09 00:43:23.187296 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 09 00:43:23.187296 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 09 00:43:23.187296 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 09 00:43:23.187296 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:23.191735 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c94a635-1c45-4dd0-a252-f8f725e3be50 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:23.191919 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 368/751] 10.222.0.33 () {56 vars in 1111 bytes} [Sun Oct 9 00:43:23 2022] GET /identity/v3/projects/service => generated 88 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:23.192601 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ab82704-92b5-4e95-b279-99b9902d9ce8 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:23.216716 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:23.254452 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:23.330242 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ab82704-92b5-4e95-b279-99b9902d9ce8 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:23.351412 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c94a635-1c45-4dd0-a252-f8f725e3be50 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:23.373345 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1997c4b-b1db-41bd-b193-76b14564ee39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:23.373978 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1997c4b-b1db-41bd-b193-76b14564ee39 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:23.374261 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1997c4b-b1db-41bd-b193-76b14564ee39 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:23.374537 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1997c4b-b1db-41bd-b193-76b14564ee39 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:23.387162 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1997c4b-b1db-41bd-b193-76b14564ee39 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:23.388638 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1997c4b-b1db-41bd-b193-76b14564ee39 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:23.394957 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-625829f4-24be-4a86-980a-b0bbb9ad2bd0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:23.395561 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-625829f4-24be-4a86-980a-b0bbb9ad2bd0 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:23.395887 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-625829f4-24be-4a86-980a-b0bbb9ad2bd0 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:23.396164 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-625829f4-24be-4a86-980a-b0bbb9ad2bd0 None admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:23.396801 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-625829f4-24be-4a86-980a-b0bbb9ad2bd0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:23.397103 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-625829f4-24be-4a86-980a-b0bbb9ad2bd0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:23.398568 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-625829f4-24be-4a86-980a-b0bbb9ad2bd0 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:23.404031 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-c1997c4b-b1db-41bd-b193-76b14564ee39 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:23.404031 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:23.404031 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:23.404031 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:23.404031 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:23.404031 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:23.404031 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:23.404031 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:23.404031 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:23.404031 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:23.404031 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:23.404031 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:23.404031 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:43:23.404031 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:43:23.404031 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:43:23.404031 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:43:23.405142 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:23.405142 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:23.405142 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:23.405142 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:23.405142 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:23.405142 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:23.405142 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:23.405142 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:23.405142 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:23.405142 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:23.405142 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:23.405142 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:23.405142 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:23.405142 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:23.405142 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:23.405142 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:23.405142 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:43:23.406232 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:43:23.406232 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:23.406232 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:23.407972 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 369/752] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:43:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:23.408902 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1997c4b-b1db-41bd-b193-76b14564ee39 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:23.412502 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 384/753] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:43:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:23.416741 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-625829f4-24be-4a86-980a-b0bbb9ad2bd0 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:23.474081 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:23.517729 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1997c4b-b1db-41bd-b193-76b14564ee39 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:23.532264 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-625829f4-24be-4a86-980a-b0bbb9ad2bd0 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:23.544464 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:23.553185 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfd2c1c8-2a15-447d-a956-f6187bda5dc2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:23.554550 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd2c1c8-2a15-447d-a956-f6187bda5dc2 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:23.555033 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd2c1c8-2a15-447d-a956-f6187bda5dc2 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:23.555456 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd2c1c8-2a15-447d-a956-f6187bda5dc2 None admin] PATH_INFO: `/v3/projects` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:23.556319 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfd2c1c8-2a15-447d-a956-f6187bda5dc2 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:23.556770 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfd2c1c8-2a15-447d-a956-f6187bda5dc2 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:23.558382 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfd2c1c8-2a15-447d-a956-f6187bda5dc2 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:23.571778 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 370/754] 10.222.0.33 () {56 vars in 1148 bytes} [Sun Oct 9 00:43:23 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:23.572773 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfd2c1c8-2a15-447d-a956-f6187bda5dc2 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:23.574316 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceaf4222-7909-4629-920c-6e31eec1084a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:23.575023 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceaf4222-7909-4629-920c-6e31eec1084a None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:23.575400 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceaf4222-7909-4629-920c-6e31eec1084a None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:23.575956 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceaf4222-7909-4629-920c-6e31eec1084a None admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:23.576831 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceaf4222-7909-4629-920c-6e31eec1084a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:23.577736 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceaf4222-7909-4629-920c-6e31eec1084a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:23.578788 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceaf4222-7909-4629-920c-6e31eec1084a None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:23.595788 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 385/755] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:43:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:23.596816 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceaf4222-7909-4629-920c-6e31eec1084a None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:23.676991 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfd2c1c8-2a15-447d-a956-f6187bda5dc2 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:23.696739 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:23.711072 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceaf4222-7909-4629-920c-6e31eec1084a None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:23.714610 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c692f11-c59f-431b-9334-1ffddb09f9bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:23.715413 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c692f11-c59f-431b-9334-1ffddb09f9bf None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:23.715885 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c692f11-c59f-431b-9334-1ffddb09f9bf None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:23.716358 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c692f11-c59f-431b-9334-1ffddb09f9bf None admin] PATH_INFO: `/v3/users/nova` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:23.723983 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c692f11-c59f-431b-9334-1ffddb09f9bf None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:23.726197 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c692f11-c59f-431b-9334-1ffddb09f9bf None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:23.738439 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-9c692f11-c59f-431b-9334-1ffddb09f9bf None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 09 00:43:23.738439 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:23.738439 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:23.738439 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:23.738439 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:23.738439 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:23.738439 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:23.738439 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:23.738439 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:23.738439 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:23.738439 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:23.738439 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:23.738439 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 09 00:43:23.738439 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 09 00:43:23.738439 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 09 00:43:23.738439 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 09 00:43:23.739537 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:23.739537 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:23.739537 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 09 00:43:23.739537 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 09 00:43:23.739537 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 09 00:43:23.739537 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 09 00:43:23.739537 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:23.739537 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:23.739537 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:23.739537 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:23.739537 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:23.739537 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:23.739537 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:23.739537 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:23.739537 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:23.739537 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:23.739537 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:23.739537 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:23.740582 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:23.740582 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:23.740582 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 09 00:43:23.740582 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 09 00:43:23.740582 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 09 00:43:23.740582 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 09 00:43:23.740582 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 09 00:43:23.740582 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 09 00:43:23.740582 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 09 00:43:23.740582 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:23.743634 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 371/756] 10.222.0.33 () {56 vars in 1093 bytes} [Sun Oct 9 00:43:23 2022] GET /identity/v3/users/nova => generated 82 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:23.744615 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c692f11-c59f-431b-9334-1ffddb09f9bf None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:23.751978 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4013931e-8c81-4880-9d5d-e60ce768249f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:23.752676 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4013931e-8c81-4880-9d5d-e60ce768249f None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:23.753022 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4013931e-8c81-4880-9d5d-e60ce768249f None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:23.753775 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4013931e-8c81-4880-9d5d-e60ce768249f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:23.754578 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4013931e-8c81-4880-9d5d-e60ce768249f None admin] RBAC: Adding query filter params (role.id=092036ee46ac427baf4d56468575f697, scope.project.id=013c83476b1340fbb051061833c4a9c6, user.id=70e367e802894b08afd7d756d8d9ecff) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:23.755300 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4013931e-8c81-4880-9d5d-e60ce768249f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:23.756788 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4013931e-8c81-4880-9d5d-e60ce768249f None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:23.764504 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 386/757] 10.222.0.33 () {56 vars in 1374 bytes} [Sun Oct 9 00:43:23 2022] GET /identity/v3/role_assignments?user.id=70e367e802894b08afd7d756d8d9ecff&scope.project.id=013c83476b1340fbb051061833c4a9c6&role.id=092036ee46ac427baf4d56468575f697 => generated 259 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:23.765495 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4013931e-8c81-4880-9d5d-e60ce768249f None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:23.799509 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:23.856993 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c692f11-c59f-431b-9334-1ffddb09f9bf None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:23.905021 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efd7aaed-8452-40af-b41f-1058f4103ec8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:23.906037 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd7aaed-8452-40af-b41f-1058f4103ec8 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:23.906523 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd7aaed-8452-40af-b41f-1058f4103ec8 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:23.907042 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd7aaed-8452-40af-b41f-1058f4103ec8 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:23.912770 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4013931e-8c81-4880-9d5d-e60ce768249f None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:23.915721 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efd7aaed-8452-40af-b41f-1058f4103ec8 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:23.917365 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efd7aaed-8452-40af-b41f-1058f4103ec8 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:23.943504 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-efd7aaed-8452-40af-b41f-1058f4103ec8 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 09 00:43:23.943504 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:23.943504 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:23.943504 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:23.943504 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:23.943504 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:23.943504 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:23.943504 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:23.943504 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:23.943504 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:23.943504 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:23.943504 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:23.943504 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 09 00:43:23.943504 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 09 00:43:23.943504 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 09 00:43:23.945272 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 09 00:43:23.945272 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:23.945272 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:23.945272 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:23.945272 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:23.945272 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:23.945272 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:23.945272 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:23.945272 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:23.945272 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:23.945272 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:23.945272 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:23.945272 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:23.945272 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:23.945272 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:23.945272 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:23.945272 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:23.946141 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 09 00:43:23.946141 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 09 00:43:23.946141 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 09 00:43:23.946141 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 09 00:43:23.946141 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 09 00:43:23.946141 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 09 00:43:23.946141 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 09 00:43:23.946141 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:23.955647 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 372/758] 10.222.0.33 () {56 vars in 1111 bytes} [Sun Oct 9 00:43:23 2022] GET /identity/v3/projects/service => generated 88 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:23.958352 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efd7aaed-8452-40af-b41f-1058f4103ec8 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:23.976874 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1f0ee9b-354c-44c5-b22a-974b62872662 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:23.977496 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f0ee9b-354c-44c5-b22a-974b62872662 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:23.977839 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f0ee9b-354c-44c5-b22a-974b62872662 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:23.978112 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f0ee9b-354c-44c5-b22a-974b62872662 None admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:23.978898 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1f0ee9b-354c-44c5-b22a-974b62872662 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:23.979206 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1f0ee9b-354c-44c5-b22a-974b62872662 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:23.981070 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1f0ee9b-354c-44c5-b22a-974b62872662 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:24.023804 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 387/759] 10.222.0.33 () {56 vars in 1133 bytes} [Sun Oct 9 00:43:23 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 257 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:24.028489 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1f0ee9b-354c-44c5-b22a-974b62872662 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:24.048053 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:24.142708 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:24.172748 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efd7aaed-8452-40af-b41f-1058f4103ec8 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:24.178435 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1f0ee9b-354c-44c5-b22a-974b62872662 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:24.240812 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-576d1166-89f4-4992-818b-067dff79aef4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:24.241890 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-576d1166-89f4-4992-818b-067dff79aef4 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:24.242156 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-576d1166-89f4-4992-818b-067dff79aef4 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:24.242422 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-576d1166-89f4-4992-818b-067dff79aef4 None admin] PATH_INFO: `/v3/projects` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:24.243122 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-576d1166-89f4-4992-818b-067dff79aef4 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:24.243429 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-576d1166-89f4-4992-818b-067dff79aef4 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:24.244884 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-576d1166-89f4-4992-818b-067dff79aef4 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:24.249532 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c14e1e8-cd4f-4ba4-ac57-591de1e06e56 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:24.250168 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c14e1e8-cd4f-4ba4-ac57-591de1e06e56 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:24.250447 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c14e1e8-cd4f-4ba4-ac57-591de1e06e56 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:24.250720 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c14e1e8-cd4f-4ba4-ac57-591de1e06e56 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:24.269259 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c14e1e8-cd4f-4ba4-ac57-591de1e06e56 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:24.270753 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c14e1e8-cd4f-4ba4-ac57-591de1e06e56 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:24.275283 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 373/760] 10.222.0.33 () {56 vars in 1148 bytes} [Sun Oct 9 00:43:23 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 317 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:43:24.281169 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-576d1166-89f4-4992-818b-067dff79aef4 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:24.287686 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-2c14e1e8-cd4f-4ba4-ac57-591de1e06e56 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:24.287686 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:24.287686 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:24.287686 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:24.287686 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:24.287686 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:24.287686 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:24.287686 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:24.287686 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:24.287686 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:24.287686 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:24.287686 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:24.287686 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:43:24.287686 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:43:24.287686 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:43:24.287686 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:43:24.288993 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:24.288993 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:24.288993 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:24.288993 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:24.288993 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:24.288993 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:24.288993 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:24.288993 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:24.288993 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:24.288993 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:24.288993 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:24.288993 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:24.288993 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:24.288993 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:24.288993 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:24.288993 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:24.288993 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:43:24.289875 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:43:24.289875 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:24.289875 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:24.289875 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 388/761] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:43:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 261 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:24.299148 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c14e1e8-cd4f-4ba4-ac57-591de1e06e56 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:24.462111 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:24.466892 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-576d1166-89f4-4992-818b-067dff79aef4 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:24.506721 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c14e1e8-cd4f-4ba4-ac57-591de1e06e56 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:24.532755 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a361f667-ec01-464d-b2c1-a373660b63b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:24.533375 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a361f667-ec01-464d-b2c1-a373660b63b7 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:24.533657 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a361f667-ec01-464d-b2c1-a373660b63b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:24.533928 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a361f667-ec01-464d-b2c1-a373660b63b7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:24.534656 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a361f667-ec01-464d-b2c1-a373660b63b7 None admin] RBAC: Adding query filter params (role.id=228232e130c5412a80d774978175d28b, scope.project.id=013c83476b1340fbb051061833c4a9c6, user.id=5f3247ddcda54394995d844c776bf43c) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:24.534963 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a361f667-ec01-464d-b2c1-a373660b63b7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:24.536436 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a361f667-ec01-464d-b2c1-a373660b63b7 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:24.542988 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 374/762] 10.222.0.33 () {56 vars in 1374 bytes} [Sun Oct 9 00:43:24 2022] GET /identity/v3/role_assignments?user.id=5f3247ddcda54394995d844c776bf43c&scope.project.id=013c83476b1340fbb051061833c4a9c6&role.id=228232e130c5412a80d774978175d28b => generated 259 bytes in 262 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:43:24.566895 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a36ecd46-18df-4910-a422-344926c845fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:24.567522 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a36ecd46-18df-4910-a422-344926c845fa None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:24.567850 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a36ecd46-18df-4910-a422-344926c845fa None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:24.568131 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a36ecd46-18df-4910-a422-344926c845fa None admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:24.568793 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a36ecd46-18df-4910-a422-344926c845fa None admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:24.569098 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a36ecd46-18df-4910-a422-344926c845fa None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:24.571137 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a36ecd46-18df-4910-a422-344926c845fa None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:24.574223 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:24.588039 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 389/763] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:43:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 289 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:24.596858 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a361f667-ec01-464d-b2c1-a373660b63b7 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:24.627572 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4078f5f7-91f2-4026-a37a-af7c298211e6 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:24.627945 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4078f5f7-91f2-4026-a37a-af7c298211e6 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:24.628143 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4078f5f7-91f2-4026-a37a-af7c298211e6 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:24.629539 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 390/764] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:43:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:43:24.635919 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d4b63c5-2524-4bc0-a7fa-95f1596c1a3e None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:24.636240 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d4b63c5-2524-4bc0-a7fa-95f1596c1a3e None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:24.636458 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d4b63c5-2524-4bc0-a7fa-95f1596c1a3e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:24.696382 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-1d4b63c5-2524-4bc0-a7fa-95f1596c1a3e None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:43:24.697586 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-1d4b63c5-2524-4bc0-a7fa-95f1596c1a3e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:43:24.737332 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a361f667-ec01-464d-b2c1-a373660b63b7 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:24.799349 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2c923c7-eda9-4c44-ba52-5e2d904a22fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:24.800030 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2c923c7-eda9-4c44-ba52-5e2d904a22fe None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:24.800240 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2c923c7-eda9-4c44-ba52-5e2d904a22fe None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:24.800590 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2c923c7-eda9-4c44-ba52-5e2d904a22fe None admin] PATH_INFO: `/v3/projects/service` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:24.808180 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2c923c7-eda9-4c44-ba52-5e2d904a22fe None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:24.809611 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2c923c7-eda9-4c44-ba52-5e2d904a22fe None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:24.825605 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-e2c923c7-eda9-4c44-ba52-5e2d904a22fe None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 09 00:43:24.825605 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:24.825605 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:24.825605 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:24.825605 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:24.825605 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:24.825605 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:24.825605 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:24.825605 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:24.825605 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:24.825605 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:24.825605 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:24.825605 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 09 00:43:24.825605 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 09 00:43:24.825605 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 09 00:43:24.826564 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 09 00:43:24.826564 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:24.826564 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:24.826564 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:24.826564 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:24.826564 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:24.826564 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:24.826564 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:24.826564 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:24.826564 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:24.826564 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:24.826564 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:24.826564 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:24.826564 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:24.826564 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:24.826564 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:24.826564 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:24.827466 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 09 00:43:24.827466 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 09 00:43:24.827466 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 09 00:43:24.827466 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 09 00:43:24.827466 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 09 00:43:24.827466 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 09 00:43:24.827466 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 09 00:43:24.827466 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:24.829611 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 375/765] 10.222.0.33 () {56 vars in 1111 bytes} [Sun Oct 9 00:43:24 2022] GET /identity/v3/projects/service => generated 88 bytes in 232 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:24.835578 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2c923c7-eda9-4c44-ba52-5e2d904a22fe None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:24.854944 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 391/766] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:43:24 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 220 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 09 00:43:24.883771 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:24.978688 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2c923c7-eda9-4c44-ba52-5e2d904a22fe None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:25.027273 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a52f855-205e-49de-9b0b-b84ac6741894 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:25.027911 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a52f855-205e-49de-9b0b-b84ac6741894 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:25.028186 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a52f855-205e-49de-9b0b-b84ac6741894 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:25.028455 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a52f855-205e-49de-9b0b-b84ac6741894 None admin] PATH_INFO: `/v3/projects` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:25.029129 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a52f855-205e-49de-9b0b-b84ac6741894 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:25.029429 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a52f855-205e-49de-9b0b-b84ac6741894 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:25.030853 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a52f855-205e-49de-9b0b-b84ac6741894 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:25.050091 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 376/767] 10.222.0.33 () {56 vars in 1148 bytes} [Sun Oct 9 00:43:24 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:25.056332 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d4b63c5-2524-4bc0-a7fa-95f1596c1a3e None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:25.192733 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d4b63c5-2524-4bc0-a7fa-95f1596c1a3e None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:25.231975 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:25.242107 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58ee5d06-9635-48b0-ad46-02f805a85503 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:25.242763 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ee5d06-9635-48b0-ad46-02f805a85503 None admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:25.243037 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ee5d06-9635-48b0-ad46-02f805a85503 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:25.243306 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ee5d06-9635-48b0-ad46-02f805a85503 None admin] PATH_INFO: `/v3/projects/013c83476b1340fbb051061833c4a9c6/users/e0a5d95969e74157bb2d2a57207e1651/roles/092036ee46ac427baf4d56468575f697` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:25.245249 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58ee5d06-9635-48b0-ad46-02f805a85503 None admin] RBAC: Authorizing `identity:create_grant(project_id=013c83476b1340fbb051061833c4a9c6, user_id=e0a5d95969e74157bb2d2a57207e1651, role_id=092036ee46ac427baf4d56468575f697)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:25.246727 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58ee5d06-9635-48b0-ad46-02f805a85503 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:25.267116 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 392/768] 10.222.0.33 () {58 vars in 1439 bytes} [Sun Oct 9 00:43:25 2022] PUT /identity/v3/projects/013c83476b1340fbb051061833c4a9c6/users/e0a5d95969e74157bb2d2a57207e1651/roles/092036ee46ac427baf4d56468575f697 => generated 0 bytes in 211 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:43:25.325532 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a52f855-205e-49de-9b0b-b84ac6741894 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:25.417021 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:25.438000 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a52f855-205e-49de-9b0b-b84ac6741894 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:25.475928 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-458f0c2e-ba50-4030-ae4c-daec7022227d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:25.476620 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-458f0c2e-ba50-4030-ae4c-daec7022227d None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:25.476948 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-458f0c2e-ba50-4030-ae4c-daec7022227d None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:25.477223 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-458f0c2e-ba50-4030-ae4c-daec7022227d None admin] PATH_INFO: `/v3/services/network` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:25.477797 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-458f0c2e-ba50-4030-ae4c-daec7022227d None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:25.479141 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-458f0c2e-ba50-4030-ae4c-daec7022227d None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:25.488557 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-458f0c2e-ba50-4030-ae4c-daec7022227d None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Oct 09 00:43:25.488557 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:25.488557 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 09 00:43:25.488557 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 09 00:43:25.488557 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 09 00:43:25.488557 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 09 00:43:25.488557 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 09 00:43:25.488557 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 09 00:43:25.488557 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 09 00:43:25.488557 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:25.488557 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 09 00:43:25.488557 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:25.488557 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:25.488557 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:25.488557 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:25.488557 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:25.488557 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:25.489590 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:25.489590 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:25.489590 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:25.489590 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:25.489590 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:25.489590 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:25.489590 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 09 00:43:25.489590 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 09 00:43:25.489590 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 09 00:43:25.489590 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 09 00:43:25.489590 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:25.489590 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:25.489590 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:25.489590 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:25.489590 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:25.489590 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:25.490798 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:25.490798 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:25.490798 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:25.490798 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:25.490798 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:25.490798 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:25.490798 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:25.490798 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:25.490798 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:25.490798 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:25.490798 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 09 00:43:25.490798 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 09 00:43:25.490798 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 09 00:43:25.490798 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:25.490798 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 377/769] 10.222.0.33 () {56 vars in 1111 bytes} [Sun Oct 9 00:43:25 2022] GET /identity/v3/services/network => generated 88 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:25.495653 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58ee5d06-9635-48b0-ad46-02f805a85503 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:25.608199 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58ee5d06-9635-48b0-ad46-02f805a85503 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:25.644361 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6e3698c-1516-42d0-a4dd-6d35a9b8ac15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:25.644971 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6e3698c-1516-42d0-a4dd-6d35a9b8ac15 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:25.645246 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6e3698c-1516-42d0-a4dd-6d35a9b8ac15 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:25.645515 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6e3698c-1516-42d0-a4dd-6d35a9b8ac15 None admin] PATH_INFO: `/v3/services` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:25.646135 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6e3698c-1516-42d0-a4dd-6d35a9b8ac15 None admin] RBAC: Adding query filter params (name=network) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:25.646431 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6e3698c-1516-42d0-a4dd-6d35a9b8ac15 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:25.647903 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6e3698c-1516-42d0-a4dd-6d35a9b8ac15 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:25.652964 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 393/770] 10.222.0.33 () {56 vars in 1112 bytes} [Sun Oct 9 00:43:25 2022] GET /identity/v3/services?name=network => generated 124 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:25.658226 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-458f0c2e-ba50-4030-ae4c-daec7022227d None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:25.784328 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-458f0c2e-ba50-4030-ae4c-daec7022227d None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:25.833467 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da5f0d5d-49de-46f5-ba77-c8ad162ff07e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:25.834080 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da5f0d5d-49de-46f5-ba77-c8ad162ff07e None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:25.834362 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da5f0d5d-49de-46f5-ba77-c8ad162ff07e None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:25.834638 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da5f0d5d-49de-46f5-ba77-c8ad162ff07e None admin] PATH_INFO: `/v3/services` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:25.835300 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da5f0d5d-49de-46f5-ba77-c8ad162ff07e None admin] RBAC: Adding query filter params (type=network) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:25.835625 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da5f0d5d-49de-46f5-ba77-c8ad162ff07e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:25.836999 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da5f0d5d-49de-46f5-ba77-c8ad162ff07e None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:25.840506 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46ef9d2a-12ae-4a18-9dc5-221d9b087e29 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:25.841002 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46ef9d2a-12ae-4a18-9dc5-221d9b087e29 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:25.841421 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46ef9d2a-12ae-4a18-9dc5-221d9b087e29 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:25.843121 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 378/772] 10.222.0.33 () {56 vars in 1112 bytes} [Sun Oct 9 00:43:25 2022] GET /identity/v3/services?type=network => generated 350 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:43:25.843319 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 394/772] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:43:25 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:43:25.848774 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da5f0d5d-49de-46f5-ba77-c8ad162ff07e None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:25.854424 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74fd23e4-eebe-4ced-9158-de3781a8445a None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:25.854705 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74fd23e4-eebe-4ced-9158-de3781a8445a None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:25.854982 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74fd23e4-eebe-4ced-9158-de3781a8445a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:25.879214 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-74fd23e4-eebe-4ced-9158-de3781a8445a None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:43:25.880340 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-74fd23e4-eebe-4ced-9158-de3781a8445a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:43:25.884313 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:25.946814 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:25.996459 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da5f0d5d-49de-46f5-ba77-c8ad162ff07e None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:25.997010 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 395/773] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:43:25 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 09 00:43:26.031137 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b288a895-66a3-49f1-9429-6bbec321e02d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:26.031906 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b288a895-66a3-49f1-9429-6bbec321e02d None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:26.032257 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b288a895-66a3-49f1-9429-6bbec321e02d None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:26.032601 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b288a895-66a3-49f1-9429-6bbec321e02d None admin] PATH_INFO: `/v3/endpoints` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:26.033378 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b288a895-66a3-49f1-9429-6bbec321e02d None admin] RBAC: Adding query filter params (interface=public, service_id=32ed0788aeb3467d80ee350e05ce8ca4, region_id=RegionOne) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:26.033820 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b288a895-66a3-49f1-9429-6bbec321e02d None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:26.035191 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b288a895-66a3-49f1-9429-6bbec321e02d None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:26.043820 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 379/774] 10.222.0.33 () {56 vars in 1251 bytes} [Sun Oct 9 00:43:25 2022] GET /identity/v3/endpoints?service_id=32ed0788aeb3467d80ee350e05ce8ca4&interface=public®ion_id=RegionOne => generated 194 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:26.049208 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74fd23e4-eebe-4ced-9158-de3781a8445a None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:26.202577 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74fd23e4-eebe-4ced-9158-de3781a8445a None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:26.248223 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f977c2f-7f7a-49fe-b4fe-b4d49e273fe0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:26.248832 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f977c2f-7f7a-49fe-b4fe-b4d49e273fe0 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:26.249114 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f977c2f-7f7a-49fe-b4fe-b4d49e273fe0 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:26.249383 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f977c2f-7f7a-49fe-b4fe-b4d49e273fe0 None admin] PATH_INFO: `/v3/services` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:26.249988 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f977c2f-7f7a-49fe-b4fe-b4d49e273fe0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:26.251356 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f977c2f-7f7a-49fe-b4fe-b4d49e273fe0 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:26.256649 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 396/775] 10.222.0.33 () {56 vars in 1087 bytes} [Sun Oct 9 00:43:26 2022] GET /identity/v3/services => generated 772 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:26.297559 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b288a895-66a3-49f1-9429-6bbec321e02d None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:26.420530 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:26.455221 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b288a895-66a3-49f1-9429-6bbec321e02d None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:26.492481 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-973ea78e-7530-4b45-9f5c-a70fc8eca120 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:26.493123 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973ea78e-7530-4b45-9f5c-a70fc8eca120 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:26.493404 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973ea78e-7530-4b45-9f5c-a70fc8eca120 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:26.493678 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973ea78e-7530-4b45-9f5c-a70fc8eca120 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:26.494264 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-973ea78e-7530-4b45-9f5c-a70fc8eca120 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:26.495669 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-973ea78e-7530-4b45-9f5c-a70fc8eca120 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:26.504698 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-973ea78e-7530-4b45-9f5c-a70fc8eca120 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 09 00:43:26.504698 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:26.504698 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 09 00:43:26.504698 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 09 00:43:26.504698 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 09 00:43:26.504698 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 09 00:43:26.504698 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 09 00:43:26.504698 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 09 00:43:26.504698 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 09 00:43:26.504698 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:26.504698 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 09 00:43:26.504698 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:26.504698 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:26.504698 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:26.504698 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:26.504698 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:26.504698 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:26.505635 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:26.505635 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:26.505635 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:26.505635 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:26.505635 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:26.505635 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:26.505635 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 09 00:43:26.505635 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 09 00:43:26.505635 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 09 00:43:26.505635 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 09 00:43:26.505635 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:26.505635 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:26.505635 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:26.505635 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:26.505635 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:26.505635 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:26.506448 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:26.506448 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:26.506448 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:26.506448 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:26.506448 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:26.506448 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:26.506448 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:26.506448 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:26.506448 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:26.506448 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:26.506448 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 09 00:43:26.506448 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 09 00:43:26.506448 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 09 00:43:26.506448 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:26.506448 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 380/776] 10.222.0.33 () {56 vars in 1129 bytes} [Sun Oct 9 00:43:26 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:26.516109 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f977c2f-7f7a-49fe-b4fe-b4d49e273fe0 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:26.547409 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d91089e-8823-42c9-9c62-3e9eb4c9c62b None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:26.547771 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d91089e-8823-42c9-9c62-3e9eb4c9c62b None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:26.547992 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d91089e-8823-42c9-9c62-3e9eb4c9c62b None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:26.549360 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 381/777] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:43:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 09 00:43:26.556111 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ecb469-3840-461d-b559-21fc8059bd8e None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:26.556617 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ecb469-3840-461d-b559-21fc8059bd8e None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:26.557044 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ecb469-3840-461d-b559-21fc8059bd8e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:26.580282 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:26.580482 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-21ecb469-3840-461d-b559-21fc8059bd8e None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:43:26.581615 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-21ecb469-3840-461d-b559-21fc8059bd8e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:43:26.667928 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 382/778] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:43:26 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 09 00:43:26.686709 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f977c2f-7f7a-49fe-b4fe-b4d49e273fe0 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:26.742841 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8249527d-b065-4a90-a05f-171a43313003 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:26.743363 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8249527d-b065-4a90-a05f-171a43313003 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:26.743659 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8249527d-b065-4a90-a05f-171a43313003 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:26.743936 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8249527d-b065-4a90-a05f-171a43313003 None admin] PATH_INFO: `/v3/services` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:26.744558 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8249527d-b065-4a90-a05f-171a43313003 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:26.745068 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8249527d-b065-4a90-a05f-171a43313003 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:26.746601 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8249527d-b065-4a90-a05f-171a43313003 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:26.754535 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 397/779] 10.222.0.33 () {56 vars in 1124 bytes} [Sun Oct 9 00:43:26 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:26.760479 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21ecb469-3840-461d-b559-21fc8059bd8e None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:26.860754 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:26.903347 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21ecb469-3840-461d-b559-21fc8059bd8e None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:26.941868 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ee9b279-9afa-4e45-9967-b01f298b2a0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:26.942474 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee9b279-9afa-4e45-9967-b01f298b2a0e None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:26.942751 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee9b279-9afa-4e45-9967-b01f298b2a0e None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:26.943021 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee9b279-9afa-4e45-9967-b01f298b2a0e None admin] PATH_INFO: `/v3/services` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:26.943664 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ee9b279-9afa-4e45-9967-b01f298b2a0e None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:26.944177 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ee9b279-9afa-4e45-9967-b01f298b2a0e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:26.945732 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ee9b279-9afa-4e45-9967-b01f298b2a0e None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:26.951203 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 383/780] 10.222.0.33 () {56 vars in 1124 bytes} [Sun Oct 9 00:43:26 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:26.956646 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8249527d-b065-4a90-a05f-171a43313003 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:27.061789 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ee9b279-9afa-4e45-9967-b01f298b2a0e None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:27.127104 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8249527d-b065-4a90-a05f-171a43313003 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:27.144496 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:27.212332 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-599c575e-6ade-48b0-a34e-18b872dcf701 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:27.212967 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-599c575e-6ade-48b0-a34e-18b872dcf701 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:27.213254 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-599c575e-6ade-48b0-a34e-18b872dcf701 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:27.213530 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-599c575e-6ade-48b0-a34e-18b872dcf701 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:27.214237 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-599c575e-6ade-48b0-a34e-18b872dcf701 None admin] RBAC: Adding query filter params (interface=public, service_id=66e5acc691cb49148d67dbbf01f2fa6b, region_id=RegionOne) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:27.214549 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-599c575e-6ade-48b0-a34e-18b872dcf701 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:27.215954 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-599c575e-6ade-48b0-a34e-18b872dcf701 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:27.227070 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ee9b279-9afa-4e45-9967-b01f298b2a0e None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:27.229871 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 398/781] 10.222.0.33 () {56 vars in 1251 bytes} [Sun Oct 9 00:43:26 2022] GET /identity/v3/endpoints?service_id=66e5acc691cb49148d67dbbf01f2fa6b&interface=public®ion_id=RegionOne => generated 194 bytes in 274 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:43:27.237054 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-599c575e-6ade-48b0-a34e-18b872dcf701 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:27.265148 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:27.275652 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e91c89-18cf-4054-86d2-34d68218e9ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:27.277188 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e91c89-18cf-4054-86d2-34d68218e9ef None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:27.277475 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e91c89-18cf-4054-86d2-34d68218e9ef None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:27.277754 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e91c89-18cf-4054-86d2-34d68218e9ef None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:27.304776 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0e91c89-18cf-4054-86d2-34d68218e9ef None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:43:27.315913 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-d0e91c89-18cf-4054-86d2-34d68218e9ef None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 09 00:43:27.315913 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:27.315913 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:27.315913 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:27.315913 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:27.315913 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:27.315913 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:27.315913 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:27.315913 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:27.315913 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:27.315913 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:27.315913 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:27.315913 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:43:27.315913 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:43:27.315913 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:43:27.315913 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:43:27.316816 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:43:27.316816 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:43:27.316816 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:43:27.316816 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:43:27.316816 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:27.316816 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:27.316816 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:27.316816 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:27.316816 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:27.316816 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:27.316816 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:27.316816 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:27.316816 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:27.316816 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:27.316816 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:27.316816 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:27.318164 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:27.318164 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:27.318164 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:27.318164 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:27.318164 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:43:27.318164 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:43:27.318164 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:43:27.318164 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:43:27.318164 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:43:27.318164 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:43:27.318164 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 09 00:43:27.318164 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:27.318164 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 384/782] 10.222.0.33 () {56 vars in 1096 bytes} [Sun Oct 9 00:43:27 2022] GET /identity/v3/roles/admin => generated 83 bytes in 257 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:27.318865 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df444e67-cb45-4ee1-94c3-2fc72a94c436 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:27.319111 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df444e67-cb45-4ee1-94c3-2fc72a94c436 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:27.320337 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df444e67-cb45-4ee1-94c3-2fc72a94c436 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:27.322382 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 385/783] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:43:27 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:43:27.325667 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df444e67-cb45-4ee1-94c3-2fc72a94c436 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:27.388017 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-599c575e-6ade-48b0-a34e-18b872dcf701 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:27.422941 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3c88903-2ad9-49c8-a59d-f381b8d61063 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:27.423542 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3c88903-2ad9-49c8-a59d-f381b8d61063 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:27.423863 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3c88903-2ad9-49c8-a59d-f381b8d61063 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:27.424136 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3c88903-2ad9-49c8-a59d-f381b8d61063 None admin] PATH_INFO: `/v3/services` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:27.424731 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3c88903-2ad9-49c8-a59d-f381b8d61063 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:27.426106 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3c88903-2ad9-49c8-a59d-f381b8d61063 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:27.432306 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 399/784] 10.222.0.33 () {56 vars in 1087 bytes} [Sun Oct 9 00:43:27 2022] GET /identity/v3/services => generated 772 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:43:27.433450 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6b570f-4a05-4a5c-b2e0-17feaa4c5c12 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:27.433741 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6b570f-4a05-4a5c-b2e0-17feaa4c5c12 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:27.434017 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6b570f-4a05-4a5c-b2e0-17feaa4c5c12 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:27.452433 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-ce6b570f-4a05-4a5c-b2e0-17feaa4c5c12 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:43:27.453772 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-ce6b570f-4a05-4a5c-b2e0-17feaa4c5c12 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:43:27.464424 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df444e67-cb45-4ee1-94c3-2fc72a94c436 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:27.512295 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:27.542871 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1dbc55d-742e-42c9-8379-4727f3dbdbd7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:27.543737 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1dbc55d-742e-42c9-8379-4727f3dbdbd7 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:27.544157 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1dbc55d-742e-42c9-8379-4727f3dbdbd7 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:27.544485 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1dbc55d-742e-42c9-8379-4727f3dbdbd7 None admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:27.545174 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1dbc55d-742e-42c9-8379-4727f3dbdbd7 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:27.545482 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1dbc55d-742e-42c9-8379-4727f3dbdbd7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:27.546880 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1dbc55d-742e-42c9-8379-4727f3dbdbd7 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:27.557252 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 386/785] 10.222.0.33 () {56 vars in 1099 bytes} [Sun Oct 9 00:43:27 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:27.562544 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1dbc55d-742e-42c9-8379-4727f3dbdbd7 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:27.572926 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 400/786] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:43:27 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 09 00:43:27.631147 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:27.689603 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1dbc55d-742e-42c9-8379-4727f3dbdbd7 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:27.741000 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a714add-6824-462b-b0a6-3edac9a15b5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:27.741690 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a714add-6824-462b-b0a6-3edac9a15b5e None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:27.741965 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a714add-6824-462b-b0a6-3edac9a15b5e None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:27.742234 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a714add-6824-462b-b0a6-3edac9a15b5e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:27.754332 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a714add-6824-462b-b0a6-3edac9a15b5e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:27.755811 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a714add-6824-462b-b0a6-3edac9a15b5e None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:27.774728 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-4a714add-6824-462b-b0a6-3edac9a15b5e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:27.774728 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:27.774728 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:27.774728 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:27.774728 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:27.774728 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:27.774728 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:27.774728 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:27.774728 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:27.774728 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:27.774728 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:27.774728 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:27.774728 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:43:27.774728 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:43:27.774728 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:43:27.774728 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:43:27.777084 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:27.777084 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:27.777084 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:27.777084 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:27.777084 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:27.777084 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:27.777084 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:27.777084 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:27.777084 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:27.777084 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:27.777084 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:27.777084 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:27.777084 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:27.777084 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:27.777084 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:27.777084 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:27.777084 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:43:27.779376 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:43:27.779376 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:27.779376 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:27.779376 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 387/787] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:43:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:27.783614 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce6b570f-4a05-4a5c-b2e0-17feaa4c5c12 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:27.896388 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce6b570f-4a05-4a5c-b2e0-17feaa4c5c12 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:27.932059 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49d84336-c66d-454d-bd20-21058bc55d8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:27.932661 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d84336-c66d-454d-bd20-21058bc55d8c None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:27.932938 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d84336-c66d-454d-bd20-21058bc55d8c None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:27.933206 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d84336-c66d-454d-bd20-21058bc55d8c None admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:27.933964 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49d84336-c66d-454d-bd20-21058bc55d8c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:27.934260 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49d84336-c66d-454d-bd20-21058bc55d8c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:27.935629 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49d84336-c66d-454d-bd20-21058bc55d8c None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:27.947845 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 401/788] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:43:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:27.953383 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a714add-6824-462b-b0a6-3edac9a15b5e None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:28.046120 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49d84336-c66d-454d-bd20-21058bc55d8c None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:28.093636 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:28.095117 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a714add-6824-462b-b0a6-3edac9a15b5e None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:28.115291 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:28.162402 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16226794-8bbf-41af-82d3-ef28a205b983 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:28.163622 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16226794-8bbf-41af-82d3-ef28a205b983 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:28.163974 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16226794-8bbf-41af-82d3-ef28a205b983 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:28.164352 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16226794-8bbf-41af-82d3-ef28a205b983 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:28.172838 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16226794-8bbf-41af-82d3-ef28a205b983 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:28.174284 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16226794-8bbf-41af-82d3-ef28a205b983 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:28.182963 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49d84336-c66d-454d-bd20-21058bc55d8c None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:28.186940 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-16226794-8bbf-41af-82d3-ef28a205b983 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 09 00:43:28.186940 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:28.186940 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:28.186940 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:28.186940 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:28.186940 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:28.186940 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:28.186940 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:28.186940 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:28.186940 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:28.186940 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:28.186940 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:28.186940 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 09 00:43:28.186940 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 09 00:43:28.186940 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 09 00:43:28.186940 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 09 00:43:28.187878 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:28.187878 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:28.187878 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 09 00:43:28.187878 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 09 00:43:28.187878 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 09 00:43:28.187878 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 09 00:43:28.187878 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:28.187878 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:28.187878 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:28.187878 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:28.187878 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:28.187878 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:28.187878 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:28.187878 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:28.187878 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:28.187878 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:28.187878 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:28.187878 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:28.188826 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:28.188826 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:28.188826 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 09 00:43:28.188826 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 09 00:43:28.188826 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 09 00:43:28.188826 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 09 00:43:28.188826 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 09 00:43:28.188826 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 09 00:43:28.188826 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 09 00:43:28.188826 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:28.188826 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 388/789] 10.222.0.33 () {56 vars in 1096 bytes} [Sun Oct 9 00:43:27 2022] GET /identity/v3/users/swift => generated 83 bytes in 235 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:28.189899 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ded53aec-30ea-4bd1-994e-575e89e2c251 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:28.189899 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ded53aec-30ea-4bd1-994e-575e89e2c251 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:28.190531 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ded53aec-30ea-4bd1-994e-575e89e2c251 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:28.191567 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 389/790] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:43:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:43:28.195573 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ded53aec-30ea-4bd1-994e-575e89e2c251 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:28.222401 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c699ae7-1a29-43b7-8503-40cc4363e632 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:28.223026 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c699ae7-1a29-43b7-8503-40cc4363e632 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:28.223306 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c699ae7-1a29-43b7-8503-40cc4363e632 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:28.223578 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c699ae7-1a29-43b7-8503-40cc4363e632 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:28.235303 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c699ae7-1a29-43b7-8503-40cc4363e632 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:43:28.240745 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-9c699ae7-1a29-43b7-8503-40cc4363e632 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 09 00:43:28.240745 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:28.240745 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:28.240745 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:28.240745 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:28.240745 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:28.240745 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:28.240745 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:28.240745 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:28.240745 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:28.240745 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:28.240745 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:28.240745 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:43:28.240745 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:43:28.240745 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:43:28.240745 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:43:28.242269 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:43:28.242269 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:43:28.242269 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:43:28.242269 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:43:28.242269 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:28.242269 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:28.242269 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:28.242269 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:28.242269 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:28.242269 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:28.242269 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:28.242269 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:28.242269 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:28.242269 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:28.242269 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:28.242269 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:28.243104 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:28.243104 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:28.243104 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:28.243104 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:28.243104 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:43:28.243104 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:43:28.243104 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:43:28.243104 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:43:28.243104 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:43:28.243104 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:43:28.243104 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 09 00:43:28.243104 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:28.243104 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 402/791] 10.222.0.33 () {56 vars in 1102 bytes} [Sun Oct 9 00:43:28 2022] GET /identity/v3/roles/service => generated 85 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:28.248038 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ba4c27-8367-4901-a85c-efce915df9ac None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:28.248320 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ba4c27-8367-4901-a85c-efce915df9ac None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:28.248645 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ba4c27-8367-4901-a85c-efce915df9ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:28.269409 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-12ba4c27-8367-4901-a85c-efce915df9ac None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:43:28.270736 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-12ba4c27-8367-4901-a85c-efce915df9ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:43:28.313046 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ded53aec-30ea-4bd1-994e-575e89e2c251 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:28.351717 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0a5c248-789f-4931-9e58-76699aa0e2ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:28.352961 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a5c248-789f-4931-9e58-76699aa0e2ce None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:28.353491 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a5c248-789f-4931-9e58-76699aa0e2ce None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:28.353964 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a5c248-789f-4931-9e58-76699aa0e2ce None admin] PATH_INFO: `/v3/users` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:28.354741 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 403/792] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:43:28 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 09 00:43:28.355146 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0a5c248-789f-4931-9e58-76699aa0e2ce None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:28.355731 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0a5c248-789f-4931-9e58-76699aa0e2ce None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:28.355853 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12ba4c27-8367-4901-a85c-efce915df9ac None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:28.357548 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0a5c248-789f-4931-9e58-76699aa0e2ce None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:28.377475 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 390/793] 10.222.0.33 () {56 vars in 1135 bytes} [Sun Oct 9 00:43:28 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:28.383311 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:28.388764 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0a5c248-789f-4931-9e58-76699aa0e2ce None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:28.443479 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:28.556643 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0a5c248-789f-4931-9e58-76699aa0e2ce None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:28.592799 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ce71297-47ba-4e0e-a9cf-75328db0e1bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:28.593861 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ce71297-47ba-4e0e-a9cf-75328db0e1bf None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:28.594236 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ce71297-47ba-4e0e-a9cf-75328db0e1bf None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:28.594604 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ce71297-47ba-4e0e-a9cf-75328db0e1bf None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:28.605327 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ce71297-47ba-4e0e-a9cf-75328db0e1bf None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:28.605481 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12ba4c27-8367-4901-a85c-efce915df9ac None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:28.607210 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ce71297-47ba-4e0e-a9cf-75328db0e1bf None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:28.625315 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-1ce71297-47ba-4e0e-a9cf-75328db0e1bf None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:28.625315 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:28.625315 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:28.625315 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:28.625315 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:28.625315 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:28.625315 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:28.625315 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:28.625315 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:28.625315 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:28.625315 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:28.625315 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:28.625315 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:43:28.625315 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:43:28.625315 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:43:28.625315 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:43:28.626425 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:28.626425 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:28.626425 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:28.626425 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:28.626425 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:28.626425 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:28.626425 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:28.626425 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:28.626425 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:28.626425 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:28.626425 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:28.626425 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:28.626425 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:28.626425 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:28.626425 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:28.626425 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:28.626425 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:43:28.627308 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:43:28.627308 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:28.627308 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:28.629062 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 391/794] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:43:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 240 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:28.635800 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ce71297-47ba-4e0e-a9cf-75328db0e1bf None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:28.707115 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b25712f-e17b-424f-855e-99c17fcd87f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:28.707687 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b25712f-e17b-424f-855e-99c17fcd87f3 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:28.708273 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b25712f-e17b-424f-855e-99c17fcd87f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:28.710780 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b25712f-e17b-424f-855e-99c17fcd87f3 None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:28.710780 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b25712f-e17b-424f-855e-99c17fcd87f3 None admin] RBAC: Adding query filter params (name=service) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:28.710780 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b25712f-e17b-424f-855e-99c17fcd87f3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:28.711075 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b25712f-e17b-424f-855e-99c17fcd87f3 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:28.720125 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 404/795] 10.222.0.33 () {56 vars in 1103 bytes} [Sun Oct 9 00:43:28 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 365 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:28.721279 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b25712f-e17b-424f-855e-99c17fcd87f3 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:28.732361 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:28.765316 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ce71297-47ba-4e0e-a9cf-75328db0e1bf None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:28.807804 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:28.811227 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-087df9fc-1f00-4c2f-b7a4-f3b2703631d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:28.812353 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-087df9fc-1f00-4c2f-b7a4-f3b2703631d6 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:28.812638 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-087df9fc-1f00-4c2f-b7a4-f3b2703631d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:28.812910 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-087df9fc-1f00-4c2f-b7a4-f3b2703631d6 None admin] PATH_INFO: `/v3/domains` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:28.813556 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-087df9fc-1f00-4c2f-b7a4-f3b2703631d6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:28.814122 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-087df9fc-1f00-4c2f-b7a4-f3b2703631d6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:28.815679 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-087df9fc-1f00-4c2f-b7a4-f3b2703631d6 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:28.830986 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b25712f-e17b-424f-855e-99c17fcd87f3 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:28.831368 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 392/796] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:43:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:43:28.832544 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-087df9fc-1f00-4c2f-b7a4-f3b2703631d6 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:28.871240 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-262293e1-5de4-40ab-a959-07282321c95b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:28.872192 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-262293e1-5de4-40ab-a959-07282321c95b None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:28.872656 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-262293e1-5de4-40ab-a959-07282321c95b None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:28.873110 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-262293e1-5de4-40ab-a959-07282321c95b None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:28.884886 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-262293e1-5de4-40ab-a959-07282321c95b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:43:28.890756 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-262293e1-5de4-40ab-a959-07282321c95b None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 09 00:43:28.890756 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:28.890756 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:28.890756 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:28.890756 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:28.890756 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:28.890756 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:28.890756 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:28.890756 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:28.890756 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:28.890756 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:28.890756 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:28.890756 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:43:28.890756 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:43:28.890756 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:43:28.890756 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:43:28.891688 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:43:28.891688 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:43:28.891688 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:43:28.891688 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:43:28.891688 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:28.891688 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:28.891688 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:28.891688 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:28.891688 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:28.891688 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:28.891688 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:28.891688 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:28.891688 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:28.891688 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:28.891688 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:28.891688 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:28.892523 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:28.892523 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:28.892523 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:28.892523 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:28.892523 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:43:28.892523 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:43:28.892523 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:43:28.892523 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:43:28.892523 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:43:28.892523 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:43:28.892523 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 09 00:43:28.892523 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:28.893146 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 405/797] 10.222.0.33 () {56 vars in 1096 bytes} [Sun Oct 9 00:43:28 2022] GET /identity/v3/roles/admin => generated 83 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:28.893973 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf9617ec-2e27-4525-ab50-c31954b180fb None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:28.894249 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf9617ec-2e27-4525-ab50-c31954b180fb None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:28.894517 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf9617ec-2e27-4525-ab50-c31954b180fb None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:28.895773 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 406/798] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:43:28 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:43:28.896444 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf9617ec-2e27-4525-ab50-c31954b180fb None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:28.942976 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-087df9fc-1f00-4c2f-b7a4-f3b2703631d6 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:28.979162 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3ebdb64-40f5-4d52-bea0-4469ef54bf15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:28.979907 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3ebdb64-40f5-4d52-bea0-4469ef54bf15 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:28.980203 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3ebdb64-40f5-4d52-bea0-4469ef54bf15 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:28.980477 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3ebdb64-40f5-4d52-bea0-4469ef54bf15 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:28.984269 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:28.993348 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3ebdb64-40f5-4d52-bea0-4469ef54bf15 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:28.994802 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3ebdb64-40f5-4d52-bea0-4469ef54bf15 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:29.000103 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf9617ec-2e27-4525-ab50-c31954b180fb None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:29.011058 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-e3ebdb64-40f5-4d52-bea0-4469ef54bf15 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:29.011058 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:29.011058 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:29.011058 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:29.011058 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:29.011058 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:29.011058 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:29.011058 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:29.011058 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:29.011058 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:29.011058 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:29.011058 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:29.011058 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:43:29.011058 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:43:29.011058 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:43:29.011058 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:43:29.012017 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:29.012017 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:29.012017 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:29.012017 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:29.012017 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:29.012017 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:29.012017 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:29.012017 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:29.012017 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:29.012017 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:29.012017 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:29.012017 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:29.012017 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:29.012017 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:29.012017 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:29.012017 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:29.012017 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:43:29.012893 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:43:29.012893 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:29.012893 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:29.012893 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 393/799] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:43:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:29.013123 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3ebdb64-40f5-4d52-bea0-4469ef54bf15 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:29.042852 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1d215db-30ff-4fff-a66e-5a8b07cb36ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:29.043645 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d215db-30ff-4fff-a66e-5a8b07cb36ad None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:29.043933 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d215db-30ff-4fff-a66e-5a8b07cb36ad None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:29.044262 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d215db-30ff-4fff-a66e-5a8b07cb36ad None admin] PATH_INFO: `/v3/projects/service` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:29.051134 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1d215db-30ff-4fff-a66e-5a8b07cb36ad None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:29.052617 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1d215db-30ff-4fff-a66e-5a8b07cb36ad None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:29.054917 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:29.065458 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-e1d215db-30ff-4fff-a66e-5a8b07cb36ad None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 09 00:43:29.065458 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:29.065458 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:29.065458 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:29.065458 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:29.065458 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:29.065458 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:29.065458 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:29.065458 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:29.065458 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:29.065458 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:29.065458 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:29.065458 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 09 00:43:29.065458 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 09 00:43:29.065458 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 09 00:43:29.066515 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 09 00:43:29.066515 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:29.066515 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:29.066515 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:29.066515 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:29.066515 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:29.066515 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:29.066515 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:29.066515 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:29.066515 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:29.066515 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:29.066515 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:29.066515 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:29.066515 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:29.066515 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:29.066515 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:29.066515 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:29.067375 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 09 00:43:29.067375 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 09 00:43:29.067375 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 09 00:43:29.067375 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 09 00:43:29.067375 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 09 00:43:29.067375 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 09 00:43:29.067375 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 09 00:43:29.067375 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:29.067375 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 407/800] 10.222.0.33 () {56 vars in 1111 bytes} [Sun Oct 9 00:43:28 2022] GET /identity/v3/projects/service => generated 88 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:29.067903 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52eca2ac-5509-4d7e-9e34-ac3b26b3b11e None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:29.067972 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52eca2ac-5509-4d7e-9e34-ac3b26b3b11e None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:29.068257 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52eca2ac-5509-4d7e-9e34-ac3b26b3b11e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:29.092125 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-52eca2ac-5509-4d7e-9e34-ac3b26b3b11e None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:43:29.093317 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-52eca2ac-5509-4d7e-9e34-ac3b26b3b11e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:43:29.131038 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3ebdb64-40f5-4d52-bea0-4469ef54bf15 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:29.171630 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 408/801] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:43:29 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 09 00:43:29.171630 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52eca2ac-5509-4d7e-9e34-ac3b26b3b11e None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:29.172740 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dec981d-206f-4e11-b26b-6a142fdfcfa0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:29.173403 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dec981d-206f-4e11-b26b-6a142fdfcfa0 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:29.173748 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dec981d-206f-4e11-b26b-6a142fdfcfa0 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:29.174111 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dec981d-206f-4e11-b26b-6a142fdfcfa0 None admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:29.174864 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dec981d-206f-4e11-b26b-6a142fdfcfa0 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:29.175229 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dec981d-206f-4e11-b26b-6a142fdfcfa0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:29.176691 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dec981d-206f-4e11-b26b-6a142fdfcfa0 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:29.190669 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 394/802] 10.222.0.33 () {56 vars in 1099 bytes} [Sun Oct 9 00:43:29 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:29.190669 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dec981d-206f-4e11-b26b-6a142fdfcfa0 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:29.252920 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:29.305793 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52eca2ac-5509-4d7e-9e34-ac3b26b3b11e None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:29.307090 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dec981d-206f-4e11-b26b-6a142fdfcfa0 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:29.340459 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:29.348855 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf4080fc-0f22-4d31-9d5d-2d39e106ecfd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:29.349320 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fee67fa7-5679-4630-93fb-ddecc33a6ca2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:29.349874 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf4080fc-0f22-4d31-9d5d-2d39e106ecfd None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:29.350107 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee67fa7-5679-4630-93fb-ddecc33a6ca2 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:29.350480 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf4080fc-0f22-4d31-9d5d-2d39e106ecfd None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:29.350629 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee67fa7-5679-4630-93fb-ddecc33a6ca2 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:29.351475 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee67fa7-5679-4630-93fb-ddecc33a6ca2 None admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:29.351732 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf4080fc-0f22-4d31-9d5d-2d39e106ecfd None admin] PATH_INFO: `/v3/projects` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:29.352383 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fee67fa7-5679-4630-93fb-ddecc33a6ca2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:29.352717 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf4080fc-0f22-4d31-9d5d-2d39e106ecfd None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:29.352845 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fee67fa7-5679-4630-93fb-ddecc33a6ca2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:29.353274 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf4080fc-0f22-4d31-9d5d-2d39e106ecfd None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:29.354397 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fee67fa7-5679-4630-93fb-ddecc33a6ca2 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:29.355006 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf4080fc-0f22-4d31-9d5d-2d39e106ecfd None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:29.374342 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 409/803] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:43:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:29.374342 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fee67fa7-5679-4630-93fb-ddecc33a6ca2 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:29.375145 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 395/804] 10.222.0.33 () {56 vars in 1148 bytes} [Sun Oct 9 00:43:29 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:29.377125 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf4080fc-0f22-4d31-9d5d-2d39e106ecfd None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:29.474848 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fee67fa7-5679-4630-93fb-ddecc33a6ca2 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:29.498700 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf4080fc-0f22-4d31-9d5d-2d39e106ecfd None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:29.508487 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e28592d7-a710-441a-b5dc-a1d0f42451ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:29.509121 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e28592d7-a710-441a-b5dc-a1d0f42451ef None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:29.509401 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e28592d7-a710-441a-b5dc-a1d0f42451ef None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:29.509669 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e28592d7-a710-441a-b5dc-a1d0f42451ef None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:29.510304 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:29.520474 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e28592d7-a710-441a-b5dc-a1d0f42451ef None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:29.521917 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e28592d7-a710-441a-b5dc-a1d0f42451ef None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:29.536374 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-e28592d7-a710-441a-b5dc-a1d0f42451ef None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:29.536374 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:29.536374 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:29.536374 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:29.536374 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:29.536374 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:29.536374 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:29.536374 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:29.536374 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:29.536374 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:29.536374 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:29.536374 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:29.536374 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:43:29.536374 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:43:29.536374 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:43:29.536374 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:43:29.537288 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:29.537288 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:29.537288 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:29.537288 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:29.537288 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:29.537288 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:29.537288 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:29.537288 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:29.537288 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:29.537288 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:29.537288 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:29.537288 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:29.537288 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:29.537288 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:29.537288 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:29.537288 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:29.537288 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:43:29.538824 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:43:29.538824 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:29.538824 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:29.538824 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 410/805] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:43:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:29.539062 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e28592d7-a710-441a-b5dc-a1d0f42451ef None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:29.543115 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-188758c4-5ab9-4dfb-b64f-ae241de0fa21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:29.543783 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-188758c4-5ab9-4dfb-b64f-ae241de0fa21 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:29.544073 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-188758c4-5ab9-4dfb-b64f-ae241de0fa21 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:29.544349 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-188758c4-5ab9-4dfb-b64f-ae241de0fa21 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:29.551740 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-188758c4-5ab9-4dfb-b64f-ae241de0fa21 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:29.553155 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-188758c4-5ab9-4dfb-b64f-ae241de0fa21 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:29.565330 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-188758c4-5ab9-4dfb-b64f-ae241de0fa21 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 09 00:43:29.565330 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:29.565330 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:29.565330 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:29.565330 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:29.565330 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:29.565330 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:29.565330 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:29.565330 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:29.565330 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:29.565330 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:29.565330 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:29.565330 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 09 00:43:29.565330 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 09 00:43:29.565330 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 09 00:43:29.565330 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 09 00:43:29.566700 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:29.566700 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:29.566700 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 09 00:43:29.566700 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 09 00:43:29.566700 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 09 00:43:29.566700 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 09 00:43:29.566700 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:29.566700 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:29.566700 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:29.566700 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:29.566700 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:29.566700 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:29.566700 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:29.566700 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:29.566700 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:29.566700 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:29.566700 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:29.566700 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:29.567628 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:29.567628 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:29.567628 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 09 00:43:29.567628 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 09 00:43:29.567628 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 09 00:43:29.567628 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 09 00:43:29.567628 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 09 00:43:29.567628 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 09 00:43:29.567628 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 09 00:43:29.567628 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:29.567628 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 396/806] 10.222.0.33 () {56 vars in 1117 bytes} [Sun Oct 9 00:43:29 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:29.567628 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-188758c4-5ab9-4dfb-b64f-ae241de0fa21 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:29.587475 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:29.646447 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e28592d7-a710-441a-b5dc-a1d0f42451ef None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:29.672251 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-188758c4-5ab9-4dfb-b64f-ae241de0fa21 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:29.683398 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87b9f9f3-a6eb-4c21-bdde-74273f9bcd91 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:29.684297 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b9f9f3-a6eb-4c21-bdde-74273f9bcd91 None admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:29.684600 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b9f9f3-a6eb-4c21-bdde-74273f9bcd91 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:29.684874 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b9f9f3-a6eb-4c21-bdde-74273f9bcd91 None admin] PATH_INFO: `/v3/projects/013c83476b1340fbb051061833c4a9c6/users/70e367e802894b08afd7d756d8d9ecff/roles/092036ee46ac427baf4d56468575f697` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:29.686793 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87b9f9f3-a6eb-4c21-bdde-74273f9bcd91 None admin] RBAC: Authorizing `identity:create_grant(project_id=013c83476b1340fbb051061833c4a9c6, user_id=70e367e802894b08afd7d756d8d9ecff, role_id=092036ee46ac427baf4d56468575f697)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:29.688264 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87b9f9f3-a6eb-4c21-bdde-74273f9bcd91 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:29.702247 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 411/807] 10.222.0.33 () {58 vars in 1439 bytes} [Sun Oct 9 00:43:29 2022] PUT /identity/v3/projects/013c83476b1340fbb051061833c4a9c6/users/70e367e802894b08afd7d756d8d9ecff/roles/092036ee46ac427baf4d56468575f697 => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:43:29.703301 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87b9f9f3-a6eb-4c21-bdde-74273f9bcd91 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:29.717923 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8efe5f1d-c1f3-41f2-b0c3-8f9edb6c6432 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:29.718678 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8efe5f1d-c1f3-41f2-b0c3-8f9edb6c6432 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:29.718956 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8efe5f1d-c1f3-41f2-b0c3-8f9edb6c6432 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:29.719226 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8efe5f1d-c1f3-41f2-b0c3-8f9edb6c6432 None admin] PATH_INFO: `/v3/services/network` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:29.719982 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8efe5f1d-c1f3-41f2-b0c3-8f9edb6c6432 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:29.721411 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8efe5f1d-c1f3-41f2-b0c3-8f9edb6c6432 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:29.732334 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-8efe5f1d-c1f3-41f2-b0c3-8f9edb6c6432 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Oct 09 00:43:29.732334 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:29.732334 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 09 00:43:29.732334 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 09 00:43:29.732334 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 09 00:43:29.732334 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 09 00:43:29.732334 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 09 00:43:29.732334 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 09 00:43:29.732334 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 09 00:43:29.732334 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:29.732334 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 09 00:43:29.732334 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:29.732334 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:29.732334 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:29.732334 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:29.732334 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:29.732334 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:29.733395 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:29.733395 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:29.733395 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:29.733395 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:29.733395 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:29.733395 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:29.733395 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 09 00:43:29.733395 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 09 00:43:29.733395 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 09 00:43:29.733395 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 09 00:43:29.733395 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:29.733395 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:29.733395 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:29.733395 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:29.733395 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:29.733395 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:29.734235 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:29.734235 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:29.734235 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:29.734235 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:29.734235 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:29.734235 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:29.734235 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:29.734235 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:29.734235 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:29.734235 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:29.734235 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 09 00:43:29.734235 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 09 00:43:29.734235 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 09 00:43:29.734235 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:29.738711 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 397/808] 10.222.0.33 () {56 vars in 1111 bytes} [Sun Oct 9 00:43:29 2022] GET /identity/v3/services/network => generated 88 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:29.739732 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8efe5f1d-c1f3-41f2-b0c3-8f9edb6c6432 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:29.861355 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:29.864878 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87b9f9f3-a6eb-4c21-bdde-74273f9bcd91 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:29.913930 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:29.922194 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98678660-063b-4333-888a-683afe06c97f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:29.922813 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98678660-063b-4333-888a-683afe06c97f None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:29.923098 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98678660-063b-4333-888a-683afe06c97f None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:29.923370 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98678660-063b-4333-888a-683afe06c97f None admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:29.924051 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98678660-063b-4333-888a-683afe06c97f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:29.924352 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98678660-063b-4333-888a-683afe06c97f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:29.925726 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98678660-063b-4333-888a-683afe06c97f None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:29.940055 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 412/809] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:43:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 237 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:29.941140 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98678660-063b-4333-888a-683afe06c97f None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:29.991068 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8efe5f1d-c1f3-41f2-b0c3-8f9edb6c6432 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:30.030497 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-984f7c0b-daf5-4a7f-9624-dd04942abb22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:30.031107 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-984f7c0b-daf5-4a7f-9624-dd04942abb22 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:30.031385 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-984f7c0b-daf5-4a7f-9624-dd04942abb22 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:30.031688 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-984f7c0b-daf5-4a7f-9624-dd04942abb22 None admin] PATH_INFO: `/v3/users` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:30.032462 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-984f7c0b-daf5-4a7f-9624-dd04942abb22 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:30.032767 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-984f7c0b-daf5-4a7f-9624-dd04942abb22 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:30.034631 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-984f7c0b-daf5-4a7f-9624-dd04942abb22 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:30.046013 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98678660-063b-4333-888a-683afe06c97f None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:30.053267 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 398/810] 10.222.0.33 () {56 vars in 1149 bytes} [Sun Oct 9 00:43:29 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 314 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:30.054346 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-984f7c0b-daf5-4a7f-9624-dd04942abb22 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:30.086317 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19a194ae-c96a-4d75-8336-78119ab2d4fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:30.087203 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a194ae-c96a-4d75-8336-78119ab2d4fe None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:30.087486 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a194ae-c96a-4d75-8336-78119ab2d4fe None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:30.087802 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a194ae-c96a-4d75-8336-78119ab2d4fe None admin] PATH_INFO: `/v3/services` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:30.088601 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19a194ae-c96a-4d75-8336-78119ab2d4fe None admin] RBAC: Adding query filter params (name=network) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:30.089230 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19a194ae-c96a-4d75-8336-78119ab2d4fe None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:30.090628 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19a194ae-c96a-4d75-8336-78119ab2d4fe None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:30.099494 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 413/811] 10.222.0.33 () {56 vars in 1112 bytes} [Sun Oct 9 00:43:29 2022] GET /identity/v3/services?name=network => generated 124 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:30.099494 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f62b6a6a-20f7-4bb8-afb6-972b00e50aee None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:30.099494 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f62b6a6a-20f7-4bb8-afb6-972b00e50aee None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:30.099494 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f62b6a6a-20f7-4bb8-afb6-972b00e50aee None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:30.099494 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 414/812] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:43:30 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:43:30.099494 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f62b6a6a-20f7-4bb8-afb6-972b00e50aee None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:30.126234 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:30.164905 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:30.174538 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-984f7c0b-daf5-4a7f-9624-dd04942abb22 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:30.208893 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-468b7a56-d47e-474a-abac-67096591878a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:30.209517 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f62b6a6a-20f7-4bb8-afb6-972b00e50aee None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:30.209627 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-468b7a56-d47e-474a-abac-67096591878a None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:30.209791 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-468b7a56-d47e-474a-abac-67096591878a None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:30.210600 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-468b7a56-d47e-474a-abac-67096591878a None admin] PATH_INFO: `/v3/users/nova` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:30.217768 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-468b7a56-d47e-474a-abac-67096591878a None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:30.219285 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-468b7a56-d47e-474a-abac-67096591878a None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:30.232938 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-468b7a56-d47e-474a-abac-67096591878a None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 09 00:43:30.232938 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:30.232938 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:30.232938 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:30.232938 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:30.232938 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:30.232938 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:30.232938 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:30.232938 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:30.232938 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:30.232938 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:30.232938 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:30.232938 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 09 00:43:30.232938 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 09 00:43:30.232938 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 09 00:43:30.232938 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 09 00:43:30.233903 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:30.233903 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:30.233903 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 09 00:43:30.233903 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 09 00:43:30.233903 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 09 00:43:30.233903 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 09 00:43:30.233903 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:30.233903 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:30.233903 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:30.233903 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:30.233903 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:30.233903 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:30.233903 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:30.233903 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:30.233903 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:30.233903 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:30.233903 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:30.233903 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:30.234852 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:30.234852 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:30.234852 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 09 00:43:30.234852 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 09 00:43:30.234852 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 09 00:43:30.234852 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 09 00:43:30.234852 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 09 00:43:30.234852 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 09 00:43:30.234852 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 09 00:43:30.234852 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:30.234852 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 399/813] 10.222.0.33 () {56 vars in 1093 bytes} [Sun Oct 9 00:43:30 2022] GET /identity/v3/users/nova => generated 82 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:30.235448 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-468b7a56-d47e-474a-abac-67096591878a None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:30.255547 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dce85b15-0651-4cf0-8dbc-0a48b731b41b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:30.256195 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dce85b15-0651-4cf0-8dbc-0a48b731b41b None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:30.256470 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dce85b15-0651-4cf0-8dbc-0a48b731b41b None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:30.256740 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dce85b15-0651-4cf0-8dbc-0a48b731b41b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:30.268288 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dce85b15-0651-4cf0-8dbc-0a48b731b41b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:30.269915 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dce85b15-0651-4cf0-8dbc-0a48b731b41b None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:30.285071 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-dce85b15-0651-4cf0-8dbc-0a48b731b41b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:30.285071 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:30.285071 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:30.285071 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:30.285071 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:30.285071 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:30.285071 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:30.285071 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:30.285071 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:30.285071 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:30.285071 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:30.285071 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:30.285071 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:43:30.285071 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:43:30.285071 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:43:30.285071 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:43:30.286032 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:30.286032 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:30.286032 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:30.286032 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:30.286032 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:30.286032 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:30.286032 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:30.286032 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:30.286032 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:30.286032 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:30.286032 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:30.286032 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:30.286032 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:30.286032 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:30.286032 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:30.286032 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:30.286032 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:43:30.286928 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:43:30.286928 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:30.286928 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:30.286928 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 415/814] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:43:30 2022] GET /identity/v3/domains/Default => generated 87 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:30.288058 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bb13139-baab-4674-9109-a8743666186f None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:30.288058 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bb13139-baab-4674-9109-a8743666186f None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:30.288058 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bb13139-baab-4674-9109-a8743666186f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:30.309033 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-8bb13139-baab-4674-9109-a8743666186f None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:43:30.310206 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-8bb13139-baab-4674-9109-a8743666186f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:43:30.350835 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-468b7a56-d47e-474a-abac-67096591878a None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:30.383719 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 416/815] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:43:30 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 09 00:43:30.383719 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bb13139-baab-4674-9109-a8743666186f None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:30.392913 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:30.395690 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-883e2b6f-6ccb-4ad1-8531-09ae5a8b4665 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:30.396510 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-883e2b6f-6ccb-4ad1-8531-09ae5a8b4665 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:30.397209 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-883e2b6f-6ccb-4ad1-8531-09ae5a8b4665 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:30.397951 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-883e2b6f-6ccb-4ad1-8531-09ae5a8b4665 None admin] PATH_INFO: `/v3/services` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:30.398723 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-883e2b6f-6ccb-4ad1-8531-09ae5a8b4665 None admin] RBAC: Adding query filter params (type=network) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:30.399215 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-883e2b6f-6ccb-4ad1-8531-09ae5a8b4665 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:30.400735 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-883e2b6f-6ccb-4ad1-8531-09ae5a8b4665 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:30.407416 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 400/816] 10.222.0.33 () {56 vars in 1112 bytes} [Sun Oct 9 00:43:30 2022] GET /identity/v3/services?type=network => generated 350 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:30.408991 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-883e2b6f-6ccb-4ad1-8531-09ae5a8b4665 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:30.435641 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:30.493686 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bb13139-baab-4674-9109-a8743666186f None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:30.528857 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca03f2ed-faef-4f26-8f68-cbace1273a52 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:30.529599 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca03f2ed-faef-4f26-8f68-cbace1273a52 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:30.529975 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca03f2ed-faef-4f26-8f68-cbace1273a52 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:30.530344 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca03f2ed-faef-4f26-8f68-cbace1273a52 None admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:30.531147 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca03f2ed-faef-4f26-8f68-cbace1273a52 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:30.531518 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca03f2ed-faef-4f26-8f68-cbace1273a52 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:30.533048 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca03f2ed-faef-4f26-8f68-cbace1273a52 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:30.544805 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-883e2b6f-6ccb-4ad1-8531-09ae5a8b4665 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:30.556345 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 417/817] 10.222.0.33 () {56 vars in 1133 bytes} [Sun Oct 9 00:43:30 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:30.556345 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca03f2ed-faef-4f26-8f68-cbace1273a52 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:30.597657 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02db8456-9825-4331-885d-07f74c096541 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:30.599396 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02db8456-9825-4331-885d-07f74c096541 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:30.599911 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02db8456-9825-4331-885d-07f74c096541 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:30.600425 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02db8456-9825-4331-885d-07f74c096541 None admin] PATH_INFO: `/v3/domains` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:30.602146 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02db8456-9825-4331-885d-07f74c096541 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:30.602709 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02db8456-9825-4331-885d-07f74c096541 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:30.604942 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02db8456-9825-4331-885d-07f74c096541 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:30.626168 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 401/818] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:43:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:30.626168 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02db8456-9825-4331-885d-07f74c096541 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:30.631387 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:30.680887 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca03f2ed-faef-4f26-8f68-cbace1273a52 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:30.715002 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17aa98da-400d-43f6-861e-9953bbe1d088 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:30.715192 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:30.715858 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17aa98da-400d-43f6-861e-9953bbe1d088 None admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:30.716387 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17aa98da-400d-43f6-861e-9953bbe1d088 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:30.716729 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17aa98da-400d-43f6-861e-9953bbe1d088 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:30.717554 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17aa98da-400d-43f6-861e-9953bbe1d088 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:30.718797 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17aa98da-400d-43f6-861e-9953bbe1d088 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:30.740750 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02db8456-9825-4331-885d-07f74c096541 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:30.751397 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 418/819] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:43:30 2022] POST /identity/v3/endpoints => generated 337 bytes in 190 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:43:30.751397 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17aa98da-400d-43f6-861e-9953bbe1d088 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:30.799806 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1057d7a-a0cd-4dc3-992f-cf5db9e8c3ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:30.800629 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1057d7a-a0cd-4dc3-992f-cf5db9e8c3ed None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:30.801001 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1057d7a-a0cd-4dc3-992f-cf5db9e8c3ed None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:30.801363 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1057d7a-a0cd-4dc3-992f-cf5db9e8c3ed None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:30.881413 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1057d7a-a0cd-4dc3-992f-cf5db9e8c3ed None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:30.882976 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1057d7a-a0cd-4dc3-992f-cf5db9e8c3ed None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:30.903073 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17aa98da-400d-43f6-861e-9953bbe1d088 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:30.929261 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:30.945444 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-911a380d-2b84-496a-9805-df87ca0597e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:30.946555 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-911a380d-2b84-496a-9805-df87ca0597e8 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:30.947279 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-911a380d-2b84-496a-9805-df87ca0597e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:30.948236 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-911a380d-2b84-496a-9805-df87ca0597e8 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:30.960768 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-911a380d-2b84-496a-9805-df87ca0597e8 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:30.962614 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-911a380d-2b84-496a-9805-df87ca0597e8 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:30.965164 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:30.975549 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-911a380d-2b84-496a-9805-df87ca0597e8 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 09 00:43:30.975549 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:30.975549 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:30.975549 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:30.975549 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:30.975549 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:30.975549 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:30.975549 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:30.975549 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:30.975549 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:30.975549 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:30.975549 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:30.975549 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 09 00:43:30.975549 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 09 00:43:30.975549 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 09 00:43:30.977045 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 09 00:43:30.977045 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:30.977045 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:30.977045 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:30.977045 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:30.977045 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:30.977045 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:30.977045 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:30.977045 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:30.977045 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:30.977045 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:30.977045 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:30.977045 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:30.977045 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:30.977045 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:30.977045 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:30.977045 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:30.977936 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 09 00:43:30.977936 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 09 00:43:30.977936 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 09 00:43:30.977936 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 09 00:43:30.977936 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 09 00:43:30.977936 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 09 00:43:30.977936 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 09 00:43:30.977936 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:30.979694 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 419/820] 10.222.0.33 () {56 vars in 1111 bytes} [Sun Oct 9 00:43:30 2022] GET /identity/v3/projects/service => generated 88 bytes in 237 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:30.980624 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-911a380d-2b84-496a-9805-df87ca0597e8 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:31.004899 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 402/821] 10.222.0.33 () {58 vars in 1282 bytes} [Sun Oct 9 00:43:30 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 382 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 09 00:43:31.006063 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1057d7a-a0cd-4dc3-992f-cf5db9e8c3ed None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:31.105118 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-911a380d-2b84-496a-9805-df87ca0597e8 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:31.111758 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1057d7a-a0cd-4dc3-992f-cf5db9e8c3ed None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:31.141393 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6364b705-f3c7-4af1-bc88-c22dc37d222f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:31.142019 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6364b705-f3c7-4af1-bc88-c22dc37d222f None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:31.142293 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6364b705-f3c7-4af1-bc88-c22dc37d222f None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:31.142563 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6364b705-f3c7-4af1-bc88-c22dc37d222f None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:31.143136 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6364b705-f3c7-4af1-bc88-c22dc37d222f None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:31.144549 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6364b705-f3c7-4af1-bc88-c22dc37d222f None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:31.147612 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eb864e2-2ebc-4d26-8ff4-80f3c0e0d524 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:31.148432 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb864e2-2ebc-4d26-8ff4-80f3c0e0d524 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:31.148887 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb864e2-2ebc-4d26-8ff4-80f3c0e0d524 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:31.149331 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb864e2-2ebc-4d26-8ff4-80f3c0e0d524 None admin] PATH_INFO: `/v3/projects` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:31.150172 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eb864e2-2ebc-4d26-8ff4-80f3c0e0d524 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:31.150662 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eb864e2-2ebc-4d26-8ff4-80f3c0e0d524 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:31.152263 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eb864e2-2ebc-4d26-8ff4-80f3c0e0d524 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:31.153238 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-6364b705-f3c7-4af1-bc88-c22dc37d222f None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 09 00:43:31.153238 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:31.153238 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 09 00:43:31.153238 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 09 00:43:31.153238 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 09 00:43:31.153238 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 09 00:43:31.153238 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 09 00:43:31.153238 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 09 00:43:31.153238 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 09 00:43:31.153238 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:31.153238 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 09 00:43:31.153238 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:31.153238 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:31.153238 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:31.153238 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:31.153238 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:31.153238 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:31.154247 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:31.154247 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:31.154247 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:31.154247 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:31.154247 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:31.154247 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:31.154247 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 09 00:43:31.154247 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 09 00:43:31.154247 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 09 00:43:31.154247 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 09 00:43:31.154247 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:31.154247 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:31.154247 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:31.154247 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:31.154247 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:31.154247 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:31.155060 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:31.155060 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:31.155060 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:31.155060 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:31.155060 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:31.155060 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:31.155060 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:31.155060 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:31.155060 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:31.155060 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:31.155060 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 09 00:43:31.155060 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 09 00:43:31.155060 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 09 00:43:31.155060 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:31.155060 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 420/822] 10.222.0.33 () {56 vars in 1129 bytes} [Sun Oct 9 00:43:30 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:31.155874 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6364b705-f3c7-4af1-bc88-c22dc37d222f None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:31.174240 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 403/823] 10.222.0.33 () {56 vars in 1148 bytes} [Sun Oct 9 00:43:31 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:31.175268 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eb864e2-2ebc-4d26-8ff4-80f3c0e0d524 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:31.205820 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:31.208737 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:31.285669 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eb864e2-2ebc-4d26-8ff4-80f3c0e0d524 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:31.323114 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a82b07a-c16a-4252-8f8b-46a17a3c2824 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:31.323764 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a82b07a-c16a-4252-8f8b-46a17a3c2824 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:31.324047 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a82b07a-c16a-4252-8f8b-46a17a3c2824 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:31.324328 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a82b07a-c16a-4252-8f8b-46a17a3c2824 None admin] PATH_INFO: `/v3/services` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:31.324967 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a82b07a-c16a-4252-8f8b-46a17a3c2824 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:31.325273 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a82b07a-c16a-4252-8f8b-46a17a3c2824 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:31.326671 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a82b07a-c16a-4252-8f8b-46a17a3c2824 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:31.335939 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 404/824] 10.222.0.33 () {56 vars in 1124 bytes} [Sun Oct 9 00:43:31 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:31.337236 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a82b07a-c16a-4252-8f8b-46a17a3c2824 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:31.380305 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6364b705-f3c7-4af1-bc88-c22dc37d222f None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:31.416572 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c15b072-7489-42c5-9201-5a3413cae441 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:31.417199 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c15b072-7489-42c5-9201-5a3413cae441 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:31.417602 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c15b072-7489-42c5-9201-5a3413cae441 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:31.417897 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c15b072-7489-42c5-9201-5a3413cae441 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:31.428402 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c15b072-7489-42c5-9201-5a3413cae441 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:31.430392 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c15b072-7489-42c5-9201-5a3413cae441 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:31.434799 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:31.441024 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a82b07a-c16a-4252-8f8b-46a17a3c2824 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:31.447559 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-7c15b072-7489-42c5-9201-5a3413cae441 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:31.447559 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:31.447559 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:31.447559 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:31.447559 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:31.447559 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:31.447559 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:31.447559 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:31.447559 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:31.447559 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:31.447559 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:31.447559 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:31.447559 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:43:31.447559 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:43:31.447559 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:43:31.447559 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:43:31.448498 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:31.448498 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:31.448498 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:31.448498 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:31.448498 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:31.448498 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:31.448498 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:31.448498 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:31.448498 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:31.448498 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:31.448498 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:31.448498 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:31.448498 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:31.448498 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:31.448498 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:31.448498 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:31.448498 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:43:31.449401 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:43:31.449401 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:31.449401 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:31.449401 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 421/825] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:43:31 2022] GET /identity/v3/domains/Default => generated 87 bytes in 294 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:31.449669 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c15b072-7489-42c5-9201-5a3413cae441 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:31.486057 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e7b04bf-e887-4a0c-a757-e7d67935fbee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:31.486634 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7b04bf-e887-4a0c-a757-e7d67935fbee None admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:31.486912 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7b04bf-e887-4a0c-a757-e7d67935fbee None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:31.487186 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7b04bf-e887-4a0c-a757-e7d67935fbee None admin] PATH_INFO: `/v3/projects/013c83476b1340fbb051061833c4a9c6/users/5f3247ddcda54394995d844c776bf43c/roles/228232e130c5412a80d774978175d28b` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:31.504047 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e7b04bf-e887-4a0c-a757-e7d67935fbee None admin] RBAC: Authorizing `identity:create_grant(project_id=013c83476b1340fbb051061833c4a9c6, user_id=5f3247ddcda54394995d844c776bf43c, role_id=228232e130c5412a80d774978175d28b)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:31.505472 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e7b04bf-e887-4a0c-a757-e7d67935fbee None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:31.519220 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 405/826] 10.222.0.33 () {58 vars in 1439 bytes} [Sun Oct 9 00:43:31 2022] PUT /identity/v3/projects/013c83476b1340fbb051061833c4a9c6/users/5f3247ddcda54394995d844c776bf43c/roles/228232e130c5412a80d774978175d28b => generated 0 bytes in 182 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:43:31.527862 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e7b04bf-e887-4a0c-a757-e7d67935fbee None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:31.566725 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:31.573986 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c15b072-7489-42c5-9201-5a3413cae441 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:31.612669 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30deeaf4-01ff-43e1-9900-a32e25548577 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:31.613279 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30deeaf4-01ff-43e1-9900-a32e25548577 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:31.613555 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30deeaf4-01ff-43e1-9900-a32e25548577 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:31.613818 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30deeaf4-01ff-43e1-9900-a32e25548577 None admin] PATH_INFO: `/v3/services` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:31.614445 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30deeaf4-01ff-43e1-9900-a32e25548577 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:31.614738 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30deeaf4-01ff-43e1-9900-a32e25548577 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:31.616864 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30deeaf4-01ff-43e1-9900-a32e25548577 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:31.622767 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 422/827] 10.222.0.33 () {56 vars in 1124 bytes} [Sun Oct 9 00:43:31 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:31.631624 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30deeaf4-01ff-43e1-9900-a32e25548577 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:31.659006 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e7b04bf-e887-4a0c-a757-e7d67935fbee None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:31.699465 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9d379fd-5a2e-491a-8911-12c9f1a0d996 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:31.700405 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9d379fd-5a2e-491a-8911-12c9f1a0d996 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:31.700724 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9d379fd-5a2e-491a-8911-12c9f1a0d996 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:31.701090 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9d379fd-5a2e-491a-8911-12c9f1a0d996 None admin] PATH_INFO: `/v3/domains` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:31.701931 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9d379fd-5a2e-491a-8911-12c9f1a0d996 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:31.702338 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9d379fd-5a2e-491a-8911-12c9f1a0d996 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:31.703863 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9d379fd-5a2e-491a-8911-12c9f1a0d996 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:31.722291 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 406/828] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:43:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:31.722780 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9d379fd-5a2e-491a-8911-12c9f1a0d996 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:31.732383 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:31.750611 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30deeaf4-01ff-43e1-9900-a32e25548577 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:31.798036 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d65d7a4c-27b0-4a81-a67d-30432d6a1a35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:31.799030 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d65d7a4c-27b0-4a81-a67d-30432d6a1a35 None admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:31.799328 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d65d7a4c-27b0-4a81-a67d-30432d6a1a35 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:31.799621 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d65d7a4c-27b0-4a81-a67d-30432d6a1a35 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:31.800577 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d65d7a4c-27b0-4a81-a67d-30432d6a1a35 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:31.801948 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d65d7a4c-27b0-4a81-a67d-30432d6a1a35 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:31.829555 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 423/829] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:43:31 2022] POST /identity/v3/endpoints => generated 346 bytes in 200 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 09 00:43:31.849482 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9d379fd-5a2e-491a-8911-12c9f1a0d996 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:31.899247 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-530a4b40-3a69-4c69-b54a-3f987512e126 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:31.900317 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-530a4b40-3a69-4c69-b54a-3f987512e126 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:31.900781 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-530a4b40-3a69-4c69-b54a-3f987512e126 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:31.901299 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-530a4b40-3a69-4c69-b54a-3f987512e126 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:31.909577 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-530a4b40-3a69-4c69-b54a-3f987512e126 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:31.911396 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-530a4b40-3a69-4c69-b54a-3f987512e126 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:31.925125 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-530a4b40-3a69-4c69-b54a-3f987512e126 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 09 00:43:31.925125 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:31.925125 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:31.925125 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:31.925125 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:31.925125 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:31.925125 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:31.925125 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:31.925125 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:31.925125 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:31.925125 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:31.925125 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:31.925125 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 09 00:43:31.925125 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 09 00:43:31.925125 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 09 00:43:31.926204 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 09 00:43:31.926204 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:31.926204 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:31.926204 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:31.926204 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:31.926204 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:31.926204 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:31.926204 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:31.926204 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:31.926204 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:31.926204 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:31.926204 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:31.926204 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:31.926204 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:31.926204 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:31.926204 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:31.926204 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:31.927101 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 09 00:43:31.927101 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 09 00:43:31.927101 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 09 00:43:31.927101 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 09 00:43:31.927101 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 09 00:43:31.927101 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 09 00:43:31.927101 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 09 00:43:31.927101 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:31.929369 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 407/830] 10.222.0.33 () {56 vars in 1111 bytes} [Sun Oct 9 00:43:31 2022] GET /identity/v3/projects/service => generated 88 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:31.937918 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d65d7a4c-27b0-4a81-a67d-30432d6a1a35 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:31.960566 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0933c4c2-ba0e-429b-af46-88f6603333ab None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:31.961090 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:31.962270 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0933c4c2-ba0e-429b-af46-88f6603333ab None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:31.962737 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0933c4c2-ba0e-429b-af46-88f6603333ab None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:31.964686 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 408/831] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:43:31 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 09 00:43:31.971898 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030e6541-84d4-40fb-ae1f-cb9c00e9b272 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:31.972407 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030e6541-84d4-40fb-ae1f-cb9c00e9b272 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:31.972972 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030e6541-84d4-40fb-ae1f-cb9c00e9b272 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:31.999615 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-030e6541-84d4-40fb-ae1f-cb9c00e9b272 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:43:32.001139 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-030e6541-84d4-40fb-ae1f-cb9c00e9b272 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:43:32.071646 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d65d7a4c-27b0-4a81-a67d-30432d6a1a35 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:32.071873 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:32.084180 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 409/832] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:43:31 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 09 00:43:32.118643 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7722011b-889d-4c60-9386-74e5bc6b220e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:32.119245 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7722011b-889d-4c60-9386-74e5bc6b220e None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:32.119527 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7722011b-889d-4c60-9386-74e5bc6b220e None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:32.119832 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7722011b-889d-4c60-9386-74e5bc6b220e None admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:32.120525 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7722011b-889d-4c60-9386-74e5bc6b220e None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:32.120833 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7722011b-889d-4c60-9386-74e5bc6b220e None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:32.122425 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7722011b-889d-4c60-9386-74e5bc6b220e None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:32.135816 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 424/833] 10.222.0.33 () {56 vars in 1148 bytes} [Sun Oct 9 00:43:31 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:32.141319 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-030e6541-84d4-40fb-ae1f-cb9c00e9b272 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:32.245800 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-030e6541-84d4-40fb-ae1f-cb9c00e9b272 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:32.281479 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66dfae5c-cebe-48d2-9e1a-d909cca301f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:32.282000 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66dfae5c-cebe-48d2-9e1a-d909cca301f6 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:32.282276 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66dfae5c-cebe-48d2-9e1a-d909cca301f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:32.282545 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66dfae5c-cebe-48d2-9e1a-d909cca301f6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:32.283268 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66dfae5c-cebe-48d2-9e1a-d909cca301f6 None admin] RBAC: Adding query filter params (role.id=092036ee46ac427baf4d56468575f697, scope.project.id=013c83476b1340fbb051061833c4a9c6, user.id=e0a5d95969e74157bb2d2a57207e1651) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:32.283568 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66dfae5c-cebe-48d2-9e1a-d909cca301f6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:32.285008 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66dfae5c-cebe-48d2-9e1a-d909cca301f6 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:32.292453 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 410/834] 10.222.0.33 () {56 vars in 1374 bytes} [Sun Oct 9 00:43:32 2022] GET /identity/v3/role_assignments?user.id=e0a5d95969e74157bb2d2a57207e1651&scope.project.id=013c83476b1340fbb051061833c4a9c6&role.id=092036ee46ac427baf4d56468575f697 => generated 608 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:32.355028 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7722011b-889d-4c60-9386-74e5bc6b220e None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:32.440751 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:32.469130 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7722011b-889d-4c60-9386-74e5bc6b220e None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:32.511788 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21790618-acd3-4f9e-8b0b-dc288492377e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:32.512469 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21790618-acd3-4f9e-8b0b-dc288492377e None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:32.512749 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21790618-acd3-4f9e-8b0b-dc288492377e None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:32.513017 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21790618-acd3-4f9e-8b0b-dc288492377e None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:32.524059 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21790618-acd3-4f9e-8b0b-dc288492377e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:43:32.528868 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-21790618-acd3-4f9e-8b0b-dc288492377e None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 09 00:43:32.528868 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:32.528868 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:32.528868 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:32.528868 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:32.528868 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:32.528868 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:32.528868 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:32.528868 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:32.528868 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:32.528868 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:32.528868 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:32.528868 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:43:32.528868 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:43:32.528868 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:43:32.528868 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:43:32.529771 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:43:32.529771 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:43:32.529771 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:43:32.529771 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:43:32.529771 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:32.529771 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:32.529771 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:32.529771 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:32.529771 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:32.529771 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:32.529771 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:32.529771 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:32.529771 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:32.529771 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:32.529771 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:32.529771 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:32.530595 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:32.530595 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:32.530595 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:32.530595 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:32.530595 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:43:32.530595 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:43:32.530595 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:43:32.530595 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:43:32.530595 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:43:32.530595 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:43:32.530595 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 09 00:43:32.530595 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:32.533088 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 425/835] 10.222.0.33 () {56 vars in 1096 bytes} [Sun Oct 9 00:43:32 2022] GET /identity/v3/roles/admin => generated 83 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:32.536089 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66dfae5c-cebe-48d2-9e1a-d909cca301f6 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:32.595859 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:32.641038 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66dfae5c-cebe-48d2-9e1a-d909cca301f6 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:32.675634 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-075e7b5c-6ce4-4afb-931f-1855c1c6f445 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:32.676258 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-075e7b5c-6ce4-4afb-931f-1855c1c6f445 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:32.676548 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-075e7b5c-6ce4-4afb-931f-1855c1c6f445 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:32.676777 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-075e7b5c-6ce4-4afb-931f-1855c1c6f445 None admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:32.677426 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-075e7b5c-6ce4-4afb-931f-1855c1c6f445 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:32.677724 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-075e7b5c-6ce4-4afb-931f-1855c1c6f445 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:32.679761 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-075e7b5c-6ce4-4afb-931f-1855c1c6f445 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:32.689535 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 411/836] 10.222.0.33 () {56 vars in 1099 bytes} [Sun Oct 9 00:43:32 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:43:32.695372 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21790618-acd3-4f9e-8b0b-dc288492377e None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:32.802742 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21790618-acd3-4f9e-8b0b-dc288492377e None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:32.840112 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb7f5fc-9010-4027-9beb-dcf8ecf54f85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:32.840752 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb7f5fc-9010-4027-9beb-dcf8ecf54f85 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:32.841024 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb7f5fc-9010-4027-9beb-dcf8ecf54f85 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:32.841290 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb7f5fc-9010-4027-9beb-dcf8ecf54f85 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:32.844326 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:32.853697 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fb7f5fc-9010-4027-9beb-dcf8ecf54f85 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:32.855208 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fb7f5fc-9010-4027-9beb-dcf8ecf54f85 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:32.870778 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-8fb7f5fc-9010-4027-9beb-dcf8ecf54f85 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:32.870778 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:32.870778 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:32.870778 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:32.870778 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:32.870778 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:32.870778 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:32.870778 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:32.870778 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:32.870778 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:32.870778 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:32.870778 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:32.870778 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:43:32.870778 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:43:32.870778 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:43:32.870778 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:43:32.871794 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:32.871794 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:32.871794 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:32.871794 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:32.871794 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:32.871794 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:32.871794 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:32.871794 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:32.871794 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:32.871794 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:32.871794 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:32.871794 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:32.871794 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:32.871794 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:32.871794 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:32.871794 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:32.871794 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:43:32.873092 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:43:32.873092 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:32.873092 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:32.873092 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 426/837] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:43:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:32.879275 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-075e7b5c-6ce4-4afb-931f-1855c1c6f445 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:32.980416 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-075e7b5c-6ce4-4afb-931f-1855c1c6f445 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:32.998598 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:33.015082 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77565de9-94de-44aa-a50d-79f2085b5a40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:33.015782 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77565de9-94de-44aa-a50d-79f2085b5a40 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:33.016124 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77565de9-94de-44aa-a50d-79f2085b5a40 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:33.016464 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77565de9-94de-44aa-a50d-79f2085b5a40 None admin] PATH_INFO: `/v3/domains` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:33.017161 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77565de9-94de-44aa-a50d-79f2085b5a40 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:33.017529 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77565de9-94de-44aa-a50d-79f2085b5a40 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:33.018941 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77565de9-94de-44aa-a50d-79f2085b5a40 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:33.030754 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 412/838] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:43:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:33.037624 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb7f5fc-9010-4027-9beb-dcf8ecf54f85 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:33.154279 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb7f5fc-9010-4027-9beb-dcf8ecf54f85 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:33.189920 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-710820e7-790d-4510-ad4f-69d4501b3d45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:33.190648 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-710820e7-790d-4510-ad4f-69d4501b3d45 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:33.191029 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-710820e7-790d-4510-ad4f-69d4501b3d45 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:33.191362 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-710820e7-790d-4510-ad4f-69d4501b3d45 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:33.199979 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-710820e7-790d-4510-ad4f-69d4501b3d45 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:33.201494 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-710820e7-790d-4510-ad4f-69d4501b3d45 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:33.212819 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-710820e7-790d-4510-ad4f-69d4501b3d45 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 09 00:43:33.212819 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:33.212819 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:33.212819 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:33.212819 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:33.212819 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:33.212819 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:33.212819 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:33.212819 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:33.212819 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:33.212819 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:33.212819 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:33.212819 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 09 00:43:33.212819 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 09 00:43:33.212819 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 09 00:43:33.212819 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 09 00:43:33.213971 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:33.213971 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:33.213971 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 09 00:43:33.213971 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 09 00:43:33.213971 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 09 00:43:33.213971 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 09 00:43:33.213971 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:33.213971 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:33.213971 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:33.213971 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:33.213971 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:33.213971 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:33.213971 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:33.213971 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:33.213971 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:33.213971 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:33.213971 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:33.213971 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:33.215140 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:33.215140 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:33.215140 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 09 00:43:33.215140 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 09 00:43:33.215140 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 09 00:43:33.215140 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 09 00:43:33.215140 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 09 00:43:33.215140 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 09 00:43:33.215140 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 09 00:43:33.215140 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:33.219632 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 427/839] 10.222.0.33 () {56 vars in 1096 bytes} [Sun Oct 9 00:43:33 2022] GET /identity/v3/users/swift => generated 83 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:33.224785 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77565de9-94de-44aa-a50d-79f2085b5a40 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:33.328293 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77565de9-94de-44aa-a50d-79f2085b5a40 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:33.364145 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b2b956-6e74-40b7-b57f-76b10259387c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:33.364765 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b2b956-6e74-40b7-b57f-76b10259387c None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:33.365111 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b2b956-6e74-40b7-b57f-76b10259387c None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:33.365445 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b2b956-6e74-40b7-b57f-76b10259387c None admin] PATH_INFO: `/v3/users` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:33.366821 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71b2b956-6e74-40b7-b57f-76b10259387c None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:33.367193 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71b2b956-6e74-40b7-b57f-76b10259387c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:33.368681 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71b2b956-6e74-40b7-b57f-76b10259387c None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:33.386629 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 413/840] 10.222.0.33 () {56 vars in 1135 bytes} [Sun Oct 9 00:43:33 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:33.392669 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-710820e7-790d-4510-ad4f-69d4501b3d45 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:33.445474 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:33.500062 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-710820e7-790d-4510-ad4f-69d4501b3d45 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:33.537366 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab30f3e8-6553-42ad-899a-acac78cda1cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:33.538749 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab30f3e8-6553-42ad-899a-acac78cda1cf None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:33.539027 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab30f3e8-6553-42ad-899a-acac78cda1cf None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:33.539296 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab30f3e8-6553-42ad-899a-acac78cda1cf None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:33.608987 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab30f3e8-6553-42ad-899a-acac78cda1cf None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:33.610486 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab30f3e8-6553-42ad-899a-acac78cda1cf None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:33.661139 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:33.719927 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 428/841] 10.222.0.33 () {58 vars in 1282 bytes} [Sun Oct 9 00:43:33 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 328 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 09 00:43:33.728763 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b2b956-6e74-40b7-b57f-76b10259387c None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:33.765663 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:33.779499 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0042600-729b-4596-8e20-b203c8088bbc None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:33.779805 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0042600-729b-4596-8e20-b203c8088bbc None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:33.780077 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0042600-729b-4596-8e20-b203c8088bbc None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:33.781384 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 429/842] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:43:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 09 00:43:33.786376 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb74d2f-d766-4a2b-aeab-58c453461e8f None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:33.786656 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb74d2f-d766-4a2b-aeab-58c453461e8f None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:33.787029 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb74d2f-d766-4a2b-aeab-58c453461e8f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:33.805962 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-1eb74d2f-d766-4a2b-aeab-58c453461e8f None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:43:33.807132 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-1eb74d2f-d766-4a2b-aeab-58c453461e8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:43:33.844190 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b2b956-6e74-40b7-b57f-76b10259387c None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:33.874997 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 430/843] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:43:33 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 09 00:43:33.885093 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cd30966-4a62-4e0e-a3b8-92fcfbe285f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:33.885960 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cd30966-4a62-4e0e-a3b8-92fcfbe285f7 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:33.886433 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cd30966-4a62-4e0e-a3b8-92fcfbe285f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:33.886899 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cd30966-4a62-4e0e-a3b8-92fcfbe285f7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:33.897586 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cd30966-4a62-4e0e-a3b8-92fcfbe285f7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:33.899260 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cd30966-4a62-4e0e-a3b8-92fcfbe285f7 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:33.914358 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-8cd30966-4a62-4e0e-a3b8-92fcfbe285f7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:33.914358 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:33.914358 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:33.914358 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:33.914358 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:33.914358 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:33.914358 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:33.914358 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:33.914358 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:33.914358 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:33.914358 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:33.914358 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:33.914358 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:43:33.914358 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:43:33.914358 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:43:33.914358 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:43:33.915244 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:33.915244 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:33.915244 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:33.915244 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:33.915244 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:33.915244 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:33.915244 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:33.915244 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:33.915244 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:33.915244 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:33.915244 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:33.915244 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:33.915244 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:33.915244 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:33.915244 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:33.915244 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:33.915244 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:43:33.916140 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:43:33.916140 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:33.916140 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:33.919657 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 414/844] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:43:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:33.922483 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eb74d2f-d766-4a2b-aeab-58c453461e8f None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:34.005535 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:34.029439 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eb74d2f-d766-4a2b-aeab-58c453461e8f None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:34.067952 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08712441-7b91-4b4a-9440-45604ad49a05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:34.068555 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08712441-7b91-4b4a-9440-45604ad49a05 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:34.068831 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08712441-7b91-4b4a-9440-45604ad49a05 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:34.069100 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08712441-7b91-4b4a-9440-45604ad49a05 None admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:34.070362 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08712441-7b91-4b4a-9440-45604ad49a05 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:34.070658 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08712441-7b91-4b4a-9440-45604ad49a05 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:34.072082 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08712441-7b91-4b4a-9440-45604ad49a05 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:34.084429 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 431/845] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:43:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:34.089991 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cd30966-4a62-4e0e-a3b8-92fcfbe285f7 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:34.153011 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5164fd1a-20f1-4787-b4ad-864b5198dad8 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:34.153338 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5164fd1a-20f1-4787-b4ad-864b5198dad8 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:34.153563 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5164fd1a-20f1-4787-b4ad-864b5198dad8 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:34.155144 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 432/846] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:43:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 09 00:43:34.159967 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3505f172-c019-450d-870e-35ef3b27453f None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:34.160257 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3505f172-c019-450d-870e-35ef3b27453f None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:34.160537 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3505f172-c019-450d-870e-35ef3b27453f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:34.183667 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-3505f172-c019-450d-870e-35ef3b27453f None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:43:34.184889 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-3505f172-c019-450d-870e-35ef3b27453f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:43:34.187492 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:34.200898 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cd30966-4a62-4e0e-a3b8-92fcfbe285f7 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:34.235153 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67863e75-aa35-4636-8172-f301bce90a93 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:34.235817 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67863e75-aa35-4636-8172-f301bce90a93 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:34.236129 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67863e75-aa35-4636-8172-f301bce90a93 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:34.236407 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67863e75-aa35-4636-8172-f301bce90a93 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:34.243794 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67863e75-aa35-4636-8172-f301bce90a93 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:34.245235 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67863e75-aa35-4636-8172-f301bce90a93 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:34.256437 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-67863e75-aa35-4636-8172-f301bce90a93 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 09 00:43:34.256437 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:34.256437 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:34.256437 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:34.256437 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:34.256437 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:34.256437 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:34.256437 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:34.256437 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:34.256437 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:34.256437 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:34.256437 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:34.256437 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 09 00:43:34.256437 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 09 00:43:34.256437 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 09 00:43:34.257499 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 09 00:43:34.257499 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:34.257499 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:34.257499 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:34.257499 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:34.257499 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:34.257499 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:34.257499 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:34.257499 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:34.257499 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:34.257499 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:34.257499 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:34.257499 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:34.257499 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:34.257499 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:34.257499 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:34.257499 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:34.258408 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 09 00:43:34.258408 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 09 00:43:34.258408 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 09 00:43:34.258408 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 09 00:43:34.258408 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 09 00:43:34.258408 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 09 00:43:34.258408 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 09 00:43:34.258408 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:34.258408 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 415/847] 10.222.0.33 () {56 vars in 1111 bytes} [Sun Oct 9 00:43:34 2022] GET /identity/v3/projects/service => generated 88 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:34.258874 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67863e75-aa35-4636-8172-f301bce90a93 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:34.271890 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 433/848] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:43:34 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 09 00:43:34.272952 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3505f172-c019-450d-870e-35ef3b27453f None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:34.364618 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:34.372590 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67863e75-aa35-4636-8172-f301bce90a93 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:34.408755 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3505f172-c019-450d-870e-35ef3b27453f None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:34.417168 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9c4fbc3-4262-4097-b73b-81fa11db0721 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:34.418135 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c4fbc3-4262-4097-b73b-81fa11db0721 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:34.418332 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c4fbc3-4262-4097-b73b-81fa11db0721 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:34.418678 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c4fbc3-4262-4097-b73b-81fa11db0721 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:34.430719 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9c4fbc3-4262-4097-b73b-81fa11db0721 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:43:34.437335 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-a9c4fbc3-4262-4097-b73b-81fa11db0721 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 09 00:43:34.437335 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:34.437335 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:34.437335 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:34.437335 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:34.437335 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:34.437335 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:34.437335 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:34.437335 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:34.437335 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:34.437335 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:34.437335 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:34.437335 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:43:34.437335 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:43:34.437335 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:43:34.437335 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:43:34.438428 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:43:34.438428 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:43:34.438428 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:43:34.438428 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:43:34.438428 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:34.438428 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:34.438428 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:34.438428 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:34.438428 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:34.438428 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:34.438428 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:34.438428 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:34.438428 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:34.438428 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:34.438428 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:34.438428 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:34.439433 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:34.439433 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:34.439433 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:34.439433 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:34.439433 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:43:34.439433 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:43:34.439433 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:43:34.439433 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:43:34.439433 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:43:34.439433 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:43:34.439433 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 09 00:43:34.439433 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:34.440855 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 416/849] 10.222.0.33 () {56 vars in 1102 bytes} [Sun Oct 9 00:43:34 2022] GET /identity/v3/roles/service => generated 85 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:34.447640 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9c4fbc3-4262-4097-b73b-81fa11db0721 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:34.459960 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e62fc36b-468b-4cb9-8684-41249169e98a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:34.460615 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e62fc36b-468b-4cb9-8684-41249169e98a None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:34.460897 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e62fc36b-468b-4cb9-8684-41249169e98a None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:34.461171 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e62fc36b-468b-4cb9-8684-41249169e98a None admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:34.461529 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:34.461863 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e62fc36b-468b-4cb9-8684-41249169e98a None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:34.462189 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e62fc36b-468b-4cb9-8684-41249169e98a None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:34.463741 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e62fc36b-468b-4cb9-8684-41249169e98a None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:34.481625 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 434/850] 10.222.0.33 () {56 vars in 1148 bytes} [Sun Oct 9 00:43:34 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:34.483193 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e62fc36b-468b-4cb9-8684-41249169e98a None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:34.563400 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9c4fbc3-4262-4097-b73b-81fa11db0721 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:34.600449 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e62fc36b-468b-4cb9-8684-41249169e98a None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:34.602165 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7abb32e2-735d-4e38-b298-2aff26f6955d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:34.602861 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7abb32e2-735d-4e38-b298-2aff26f6955d None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:34.603141 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7abb32e2-735d-4e38-b298-2aff26f6955d None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:34.603418 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7abb32e2-735d-4e38-b298-2aff26f6955d None admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:34.604120 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7abb32e2-735d-4e38-b298-2aff26f6955d None admin] RBAC: Adding query filter params (name=service) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:34.604418 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7abb32e2-735d-4e38-b298-2aff26f6955d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:34.605803 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7abb32e2-735d-4e38-b298-2aff26f6955d None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:34.615768 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 417/851] 10.222.0.33 () {56 vars in 1103 bytes} [Sun Oct 9 00:43:34 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:43:34.615768 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7abb32e2-735d-4e38-b298-2aff26f6955d None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:34.633866 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:34.643065 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9794842-8cea-46c4-a708-93071fef0d87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:34.643690 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9794842-8cea-46c4-a708-93071fef0d87 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:34.643969 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9794842-8cea-46c4-a708-93071fef0d87 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:34.644236 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9794842-8cea-46c4-a708-93071fef0d87 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:34.644944 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9794842-8cea-46c4-a708-93071fef0d87 None admin] RBAC: Adding query filter params (role.id=092036ee46ac427baf4d56468575f697, scope.project.id=013c83476b1340fbb051061833c4a9c6, user.id=70e367e802894b08afd7d756d8d9ecff) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:34.645247 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9794842-8cea-46c4-a708-93071fef0d87 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:34.646647 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9794842-8cea-46c4-a708-93071fef0d87 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:34.653061 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 435/852] 10.222.0.33 () {56 vars in 1374 bytes} [Sun Oct 9 00:43:34 2022] GET /identity/v3/role_assignments?user.id=70e367e802894b08afd7d756d8d9ecff&scope.project.id=013c83476b1340fbb051061833c4a9c6&role.id=092036ee46ac427baf4d56468575f697 => generated 608 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:43:34.653061 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9794842-8cea-46c4-a708-93071fef0d87 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:34.715069 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:34.751905 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7abb32e2-735d-4e38-b298-2aff26f6955d None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:34.773696 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9794842-8cea-46c4-a708-93071fef0d87 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:34.793367 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc451870-83a7-4c9e-9cc1-145b6f4800b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:34.794109 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc451870-83a7-4c9e-9cc1-145b6f4800b0 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:34.794454 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc451870-83a7-4c9e-9cc1-145b6f4800b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:34.794871 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc451870-83a7-4c9e-9cc1-145b6f4800b0 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:34.795468 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc451870-83a7-4c9e-9cc1-145b6f4800b0 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:34.796999 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc451870-83a7-4c9e-9cc1-145b6f4800b0 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:34.806588 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-dc451870-83a7-4c9e-9cc1-145b6f4800b0 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 09 00:43:34.806588 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:34.806588 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 09 00:43:34.806588 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 09 00:43:34.806588 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 09 00:43:34.806588 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 09 00:43:34.806588 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 09 00:43:34.806588 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 09 00:43:34.806588 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 09 00:43:34.806588 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:34.806588 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 09 00:43:34.806588 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:34.806588 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:34.806588 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:34.806588 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:34.806588 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:34.806588 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:34.807649 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:34.807649 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:34.807649 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:34.807649 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:34.807649 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:34.807649 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:34.807649 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 09 00:43:34.807649 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 09 00:43:34.807649 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 09 00:43:34.807649 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 09 00:43:34.807649 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:34.807649 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:34.807649 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:34.807649 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:34.807649 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:34.807649 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:34.808469 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:34.808469 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:34.808469 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:34.808469 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:34.808469 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:34.808469 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:34.808469 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:34.808469 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:34.808469 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:34.808469 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:34.808469 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 09 00:43:34.808469 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 09 00:43:34.808469 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 09 00:43:34.808469 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:34.808469 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 418/853] 10.222.0.33 () {56 vars in 1114 bytes} [Sun Oct 9 00:43:34 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:34.810170 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc5e187b-ccd7-4ab3-b6d7-4ac12fcdfcb8 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:34.810529 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc5e187b-ccd7-4ab3-b6d7-4ac12fcdfcb8 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:34.810868 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc5e187b-ccd7-4ab3-b6d7-4ac12fcdfcb8 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:34.812396 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 419/854] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:43:34 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:43:34.815236 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc5e187b-ccd7-4ab3-b6d7-4ac12fcdfcb8 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:34.818879 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bed1087-0810-437d-8ccf-c770ffc7fc35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:34.819554 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bed1087-0810-437d-8ccf-c770ffc7fc35 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:34.819862 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bed1087-0810-437d-8ccf-c770ffc7fc35 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:34.820136 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bed1087-0810-437d-8ccf-c770ffc7fc35 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:34.833063 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bed1087-0810-437d-8ccf-c770ffc7fc35 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:34.834546 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bed1087-0810-437d-8ccf-c770ffc7fc35 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:34.851717 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-6bed1087-0810-437d-8ccf-c770ffc7fc35 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:34.851717 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:34.851717 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:34.851717 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:34.851717 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:34.851717 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:34.851717 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:34.851717 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:34.851717 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:34.851717 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:34.851717 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:34.851717 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:34.851717 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:43:34.851717 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:43:34.851717 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:43:34.851717 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:43:34.852669 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:34.852669 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:34.852669 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:34.852669 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:34.852669 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:34.852669 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:34.852669 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:34.852669 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:34.852669 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:34.852669 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:34.852669 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:34.852669 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:34.852669 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:34.852669 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:34.852669 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:34.852669 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:34.852669 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:43:34.853543 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:43:34.853543 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:34.853543 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:34.853543 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 436/855] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:43:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:34.853999 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71261caa-1f5c-471a-ba13-34f756b237a8 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:34.854279 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71261caa-1f5c-471a-ba13-34f756b237a8 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:34.855207 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71261caa-1f5c-471a-ba13-34f756b237a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:34.875824 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-71261caa-1f5c-471a-ba13-34f756b237a8 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:43:34.877017 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-71261caa-1f5c-471a-ba13-34f756b237a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:43:34.921627 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:34.939406 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc5e187b-ccd7-4ab3-b6d7-4ac12fcdfcb8 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:34.959621 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 437/856] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:43:34 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 09 00:43:34.959621 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71261caa-1f5c-471a-ba13-34f756b237a8 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:34.990661 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ba312ec-ad6f-4bfc-bbe6-b47eba3abe87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:34.991337 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba312ec-ad6f-4bfc-bbe6-b47eba3abe87 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:34.991926 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba312ec-ad6f-4bfc-bbe6-b47eba3abe87 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:34.992208 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba312ec-ad6f-4bfc-bbe6-b47eba3abe87 None admin] PATH_INFO: `/v3/services` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:34.993275 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ba312ec-ad6f-4bfc-bbe6-b47eba3abe87 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:34.993580 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ba312ec-ad6f-4bfc-bbe6-b47eba3abe87 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:34.995494 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ba312ec-ad6f-4bfc-bbe6-b47eba3abe87 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:34.997291 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:35.002845 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 420/857] 10.222.0.33 () {56 vars in 1114 bytes} [Sun Oct 9 00:43:34 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:43:35.008805 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ba312ec-ad6f-4bfc-bbe6-b47eba3abe87 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:35.090459 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71261caa-1f5c-471a-ba13-34f756b237a8 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:35.125826 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55d227ad-7f27-4eca-9c43-9e8465024223 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:35.126746 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55d227ad-7f27-4eca-9c43-9e8465024223 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:35.127215 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55d227ad-7f27-4eca-9c43-9e8465024223 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:35.127689 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55d227ad-7f27-4eca-9c43-9e8465024223 None admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:35.128504 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55d227ad-7f27-4eca-9c43-9e8465024223 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:35.128981 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55d227ad-7f27-4eca-9c43-9e8465024223 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:35.129102 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ba312ec-ad6f-4bfc-bbe6-b47eba3abe87 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:35.130657 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55d227ad-7f27-4eca-9c43-9e8465024223 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:35.143710 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 438/858] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:43:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:35.151500 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55d227ad-7f27-4eca-9c43-9e8465024223 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:35.167025 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e53a702d-1776-4b10-81e3-10e760851d42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:35.167662 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e53a702d-1776-4b10-81e3-10e760851d42 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:35.167943 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e53a702d-1776-4b10-81e3-10e760851d42 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:35.168214 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e53a702d-1776-4b10-81e3-10e760851d42 None admin] PATH_INFO: `/v3/services` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:35.169106 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e53a702d-1776-4b10-81e3-10e760851d42 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:35.169413 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e53a702d-1776-4b10-81e3-10e760851d42 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:35.171252 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e53a702d-1776-4b10-81e3-10e760851d42 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:35.176660 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 421/859] 10.222.0.33 () {56 vars in 1114 bytes} [Sun Oct 9 00:43:35 2022] GET /identity/v3/services?type=volumev3 => generated 125 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:43:35.199564 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:35.231576 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e53a702d-1776-4b10-81e3-10e760851d42 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:35.293625 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55d227ad-7f27-4eca-9c43-9e8465024223 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:35.316826 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:35.332311 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e4e3199-464f-425f-ac14-663ae24c18a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:35.333353 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4e3199-464f-425f-ac14-663ae24c18a5 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:35.333867 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4e3199-464f-425f-ac14-663ae24c18a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:35.334342 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4e3199-464f-425f-ac14-663ae24c18a5 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:35.343875 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e4e3199-464f-425f-ac14-663ae24c18a5 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:35.345736 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e4e3199-464f-425f-ac14-663ae24c18a5 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:35.360702 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-1e4e3199-464f-425f-ac14-663ae24c18a5 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 09 00:43:35.360702 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:35.360702 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:35.360702 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:35.360702 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:35.360702 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:35.360702 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:35.360702 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:35.360702 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:35.360702 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:35.360702 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:35.360702 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:35.360702 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 09 00:43:35.360702 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 09 00:43:35.360702 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 09 00:43:35.360702 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 09 00:43:35.361705 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:35.361705 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:35.361705 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 09 00:43:35.361705 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 09 00:43:35.361705 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 09 00:43:35.361705 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 09 00:43:35.361705 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:35.361705 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:35.361705 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:35.361705 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:35.361705 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:35.361705 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:35.361705 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:35.361705 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:35.361705 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:35.361705 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:35.361705 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:35.361705 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:35.362682 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:35.362682 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:35.362682 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 09 00:43:35.362682 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 09 00:43:35.362682 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 09 00:43:35.362682 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 09 00:43:35.362682 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 09 00:43:35.362682 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 09 00:43:35.362682 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 09 00:43:35.362682 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:35.362682 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 439/860] 10.222.0.33 () {56 vars in 1117 bytes} [Sun Oct 9 00:43:35 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:35.365094 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e53a702d-1776-4b10-81e3-10e760851d42 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:35.367490 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e4e3199-464f-425f-ac14-663ae24c18a5 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:35.406129 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54d09d26-6cf6-44f9-b562-0d8fca92fbd2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:35.406770 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d09d26-6cf6-44f9-b562-0d8fca92fbd2 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:35.407051 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d09d26-6cf6-44f9-b562-0d8fca92fbd2 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:35.407380 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d09d26-6cf6-44f9-b562-0d8fca92fbd2 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:35.407962 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54d09d26-6cf6-44f9-b562-0d8fca92fbd2 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:35.409314 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54d09d26-6cf6-44f9-b562-0d8fca92fbd2 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:35.418964 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-54d09d26-6cf6-44f9-b562-0d8fca92fbd2 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 09 00:43:35.418964 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:35.418964 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 09 00:43:35.418964 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 09 00:43:35.418964 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 09 00:43:35.418964 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 09 00:43:35.418964 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 09 00:43:35.418964 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 09 00:43:35.418964 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 09 00:43:35.418964 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:35.418964 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 09 00:43:35.418964 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:35.418964 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:35.418964 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:35.418964 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:35.418964 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:35.420097 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:35.420097 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:35.420097 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:35.420097 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:35.420097 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:35.420097 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:35.420097 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:35.420097 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 09 00:43:35.420097 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 09 00:43:35.420097 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 09 00:43:35.420097 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 09 00:43:35.420097 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:35.420097 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:35.420097 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:35.420097 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:35.420097 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:35.420097 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:35.420988 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:35.420988 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:35.420988 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:35.420988 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:35.420988 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:35.420988 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:35.420988 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:35.420988 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:35.420988 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:35.420988 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:35.420988 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 09 00:43:35.420988 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 09 00:43:35.420988 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 09 00:43:35.420988 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:35.420988 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 422/861] 10.222.0.33 () {56 vars in 1132 bytes} [Sun Oct 9 00:43:35 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 09 00:43:35.427622 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54d09d26-6cf6-44f9-b562-0d8fca92fbd2 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:35.487852 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:35.496224 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e4e3199-464f-425f-ac14-663ae24c18a5 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:35.534821 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0e69c30-9bc2-45f4-9997-35e2f1cff9ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:35.535485 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e69c30-9bc2-45f4-9997-35e2f1cff9ea None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:35.536169 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e69c30-9bc2-45f4-9997-35e2f1cff9ea None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:35.536169 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e69c30-9bc2-45f4-9997-35e2f1cff9ea None admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:35.536864 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0e69c30-9bc2-45f4-9997-35e2f1cff9ea None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:35.537170 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0e69c30-9bc2-45f4-9997-35e2f1cff9ea None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:35.539535 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0e69c30-9bc2-45f4-9997-35e2f1cff9ea None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:35.558098 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 440/862] 10.222.0.33 () {56 vars in 1149 bytes} [Sun Oct 9 00:43:35 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:43:35.562861 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0e69c30-9bc2-45f4-9997-35e2f1cff9ea None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:35.570569 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54d09d26-6cf6-44f9-b562-0d8fca92fbd2 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:35.598900 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:35.610307 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa291649-258f-4229-96de-a0671a1d3ec3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:35.611289 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa291649-258f-4229-96de-a0671a1d3ec3 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:35.611821 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa291649-258f-4229-96de-a0671a1d3ec3 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:35.612287 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa291649-258f-4229-96de-a0671a1d3ec3 None admin] PATH_INFO: `/v3/services` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:35.613155 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa291649-258f-4229-96de-a0671a1d3ec3 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:35.613672 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa291649-258f-4229-96de-a0671a1d3ec3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:35.615230 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa291649-258f-4229-96de-a0671a1d3ec3 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:35.621166 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 423/863] 10.222.0.33 () {56 vars in 1126 bytes} [Sun Oct 9 00:43:35 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:35.626789 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa291649-258f-4229-96de-a0671a1d3ec3 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:35.688630 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0e69c30-9bc2-45f4-9997-35e2f1cff9ea None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:35.728392 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a2eefa3-afe8-42f2-8f75-67bb9da01b9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:35.729246 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a2eefa3-afe8-42f2-8f75-67bb9da01b9b None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:35.729699 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a2eefa3-afe8-42f2-8f75-67bb9da01b9b None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:35.730120 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a2eefa3-afe8-42f2-8f75-67bb9da01b9b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:35.756589 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa291649-258f-4229-96de-a0671a1d3ec3 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:35.762268 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:35.795292 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdb77107-4f47-4404-9b48-c50472394533 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:35.796095 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdb77107-4f47-4404-9b48-c50472394533 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:35.796478 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdb77107-4f47-4404-9b48-c50472394533 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:35.796840 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdb77107-4f47-4404-9b48-c50472394533 None admin] PATH_INFO: `/v3/services` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:35.797559 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdb77107-4f47-4404-9b48-c50472394533 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:35.797955 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdb77107-4f47-4404-9b48-c50472394533 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:35.799696 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdb77107-4f47-4404-9b48-c50472394533 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:35.811707 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 424/864] 10.222.0.33 () {56 vars in 1126 bytes} [Sun Oct 9 00:43:35 2022] GET /identity/v3/services?type=compute_legacy => generated 131 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:43:35.830177 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a2eefa3-afe8-42f2-8f75-67bb9da01b9b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:35.831675 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a2eefa3-afe8-42f2-8f75-67bb9da01b9b None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:35.942140 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 441/865] 10.222.0.33 () {58 vars in 1282 bytes} [Sun Oct 9 00:43:35 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 379 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Oct 09 00:43:35.948385 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdb77107-4f47-4404-9b48-c50472394533 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:36.018199 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:36.067492 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdb77107-4f47-4404-9b48-c50472394533 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:36.105896 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f84b80f3-8bc3-46c2-be02-b4412a6c11a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:36.106859 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f84b80f3-8bc3-46c2-be02-b4412a6c11a6 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:36.107367 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f84b80f3-8bc3-46c2-be02-b4412a6c11a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:36.107873 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f84b80f3-8bc3-46c2-be02-b4412a6c11a6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:36.120004 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f84b80f3-8bc3-46c2-be02-b4412a6c11a6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:36.121857 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f84b80f3-8bc3-46c2-be02-b4412a6c11a6 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:36.138672 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-f84b80f3-8bc3-46c2-be02-b4412a6c11a6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:36.138672 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:36.138672 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:36.138672 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:36.138672 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:36.138672 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:36.138672 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:36.138672 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:36.138672 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:36.138672 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:36.138672 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:36.138672 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:36.138672 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:43:36.138672 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:43:36.138672 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:43:36.138672 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:43:36.139672 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:36.139672 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:36.139672 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:36.139672 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:36.139672 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:36.139672 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:36.139672 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:36.139672 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:36.139672 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:36.139672 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:36.139672 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:36.139672 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:36.139672 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:36.139672 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:36.139672 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:36.139672 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:36.139672 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:43:36.140546 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:43:36.140546 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:36.140546 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:36.142622 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 425/866] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:43:35 2022] GET /identity/v3/domains/Default => generated 87 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:36.147861 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a2eefa3-afe8-42f2-8f75-67bb9da01b9b None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:36.202518 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:36.251577 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a2eefa3-afe8-42f2-8f75-67bb9da01b9b None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:36.286811 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ce12bce-49c2-4955-b875-1b94ca9a0a68 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:36.287461 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce12bce-49c2-4955-b875-1b94ca9a0a68 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:36.287767 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce12bce-49c2-4955-b875-1b94ca9a0a68 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:36.288067 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce12bce-49c2-4955-b875-1b94ca9a0a68 None admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:36.289014 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ce12bce-49c2-4955-b875-1b94ca9a0a68 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:36.289307 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ce12bce-49c2-4955-b875-1b94ca9a0a68 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:36.290707 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ce12bce-49c2-4955-b875-1b94ca9a0a68 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:36.302456 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 442/867] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:43:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:36.308862 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f84b80f3-8bc3-46c2-be02-b4412a6c11a6 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:36.418284 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f84b80f3-8bc3-46c2-be02-b4412a6c11a6 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:36.425741 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:36.455315 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca6ef99d-034d-4439-a29e-c6eda4985f05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:36.455974 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca6ef99d-034d-4439-a29e-c6eda4985f05 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:36.456251 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca6ef99d-034d-4439-a29e-c6eda4985f05 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:36.456522 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca6ef99d-034d-4439-a29e-c6eda4985f05 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:36.464155 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca6ef99d-034d-4439-a29e-c6eda4985f05 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:36.465595 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca6ef99d-034d-4439-a29e-c6eda4985f05 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:36.477124 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-ca6ef99d-034d-4439-a29e-c6eda4985f05 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 09 00:43:36.477124 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:36.477124 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:36.477124 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:36.477124 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:36.477124 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:36.477124 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:36.477124 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:36.477124 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:36.477124 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:36.477124 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:36.477124 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:36.477124 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 09 00:43:36.477124 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 09 00:43:36.477124 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 09 00:43:36.478050 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 09 00:43:36.478050 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:36.478050 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:36.478050 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:36.478050 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:36.478050 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:36.478050 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:36.478050 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:36.478050 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:36.478050 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:36.478050 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:36.478050 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:36.478050 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:36.478050 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:36.478050 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:36.478050 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:36.478050 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:36.478933 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 09 00:43:36.478933 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 09 00:43:36.478933 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 09 00:43:36.478933 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 09 00:43:36.478933 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 09 00:43:36.478933 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 09 00:43:36.478933 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 09 00:43:36.478933 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:36.478933 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 426/868] 10.222.0.33 () {56 vars in 1111 bytes} [Sun Oct 9 00:43:36 2022] GET /identity/v3/projects/service => generated 88 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:36.484295 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ce12bce-49c2-4955-b875-1b94ca9a0a68 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:36.590877 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ce12bce-49c2-4955-b875-1b94ca9a0a68 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:36.604098 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:36.626628 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa09926c-5e3c-45c9-a9eb-afe9fd8f3b2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:36.627234 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa09926c-5e3c-45c9-a9eb-afe9fd8f3b2f None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:36.627562 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa09926c-5e3c-45c9-a9eb-afe9fd8f3b2f None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:36.627869 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa09926c-5e3c-45c9-a9eb-afe9fd8f3b2f None admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:36.628563 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa09926c-5e3c-45c9-a9eb-afe9fd8f3b2f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:36.628870 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa09926c-5e3c-45c9-a9eb-afe9fd8f3b2f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:36.630296 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa09926c-5e3c-45c9-a9eb-afe9fd8f3b2f None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:36.643406 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 443/869] 10.222.0.33 () {56 vars in 1148 bytes} [Sun Oct 9 00:43:36 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:36.649886 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca6ef99d-034d-4439-a29e-c6eda4985f05 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:36.752807 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca6ef99d-034d-4439-a29e-c6eda4985f05 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:36.791112 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a43e50e6-a7a9-4f7c-842b-8f1214ce122e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:36.792587 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a43e50e6-a7a9-4f7c-842b-8f1214ce122e None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:36.793085 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a43e50e6-a7a9-4f7c-842b-8f1214ce122e None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:36.793553 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a43e50e6-a7a9-4f7c-842b-8f1214ce122e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:36.794496 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a43e50e6-a7a9-4f7c-842b-8f1214ce122e None admin] RBAC: Adding query filter params (role.id=228232e130c5412a80d774978175d28b, scope.project.id=013c83476b1340fbb051061833c4a9c6, user.id=5f3247ddcda54394995d844c776bf43c) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:36.795012 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a43e50e6-a7a9-4f7c-842b-8f1214ce122e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:36.796704 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a43e50e6-a7a9-4f7c-842b-8f1214ce122e None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:36.804708 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 427/870] 10.222.0.33 () {56 vars in 1374 bytes} [Sun Oct 9 00:43:36 2022] GET /identity/v3/role_assignments?user.id=5f3247ddcda54394995d844c776bf43c&scope.project.id=013c83476b1340fbb051061833c4a9c6&role.id=228232e130c5412a80d774978175d28b => generated 608 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:36.904324 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee55c97-ed77-4b3b-9677-b66a146e51b7 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:36.904651 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee55c97-ed77-4b3b-9677-b66a146e51b7 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:36.904868 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee55c97-ed77-4b3b-9677-b66a146e51b7 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:36.906499 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 444/871] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:43:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:43:36.912201 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8676af5-1686-44fc-9aae-42789d2e339a None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:36.912766 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8676af5-1686-44fc-9aae-42789d2e339a None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:36.914040 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8676af5-1686-44fc-9aae-42789d2e339a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:36.933567 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-d8676af5-1686-44fc-9aae-42789d2e339a None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:43:36.935156 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-d8676af5-1686-44fc-9aae-42789d2e339a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:43:36.963032 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:37.009737 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 428/872] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:43:36 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 09 00:43:37.278932 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dee55c97-ed77-4b3b-9677-b66a146e51b7 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:37.388836 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dee55c97-ed77-4b3b-9677-b66a146e51b7 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:37.424655 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2478370-5c2e-4c12-92ba-82724a6cfe42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:37.425283 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2478370-5c2e-4c12-92ba-82724a6cfe42 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:37.425563 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2478370-5c2e-4c12-92ba-82724a6cfe42 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:37.425836 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2478370-5c2e-4c12-92ba-82724a6cfe42 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:37.426465 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2478370-5c2e-4c12-92ba-82724a6cfe42 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:37.427852 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2478370-5c2e-4c12-92ba-82724a6cfe42 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:37.437474 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-a2478370-5c2e-4c12-92ba-82724a6cfe42 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 09 00:43:37.437474 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:37.437474 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 09 00:43:37.437474 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 09 00:43:37.437474 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 09 00:43:37.437474 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 09 00:43:37.437474 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 09 00:43:37.437474 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 09 00:43:37.437474 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 09 00:43:37.437474 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:37.437474 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 09 00:43:37.437474 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:37.437474 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:37.437474 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:37.437474 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:37.437474 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:37.437474 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:37.438421 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:37.438421 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:37.438421 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:37.438421 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:37.438421 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:37.438421 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:37.438421 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 09 00:43:37.438421 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 09 00:43:37.438421 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 09 00:43:37.438421 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 09 00:43:37.438421 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:37.438421 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:37.438421 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:37.438421 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:37.438421 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:37.438421 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:37.439236 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:37.439236 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:37.439236 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:37.439236 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:37.439236 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:37.439236 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:37.439236 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:37.439236 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:37.439236 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:37.439236 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:37.439236 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 09 00:43:37.439236 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 09 00:43:37.439236 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 09 00:43:37.439236 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:37.441975 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 445/873] 10.222.0.33 () {56 vars in 1126 bytes} [Sun Oct 9 00:43:37 2022] GET /identity/v3/services/object-store => generated 93 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:37.443970 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02090782-7b8b-4866-a2ca-bd44fcbb0f42 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:37.444993 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02090782-7b8b-4866-a2ca-bd44fcbb0f42 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:37.445165 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2478370-5c2e-4c12-92ba-82724a6cfe42 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:37.445637 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02090782-7b8b-4866-a2ca-bd44fcbb0f42 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:37.447340 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 429/874] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:43:37 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:43:37.452627 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dacd9699-89c5-4212-883c-5f62b4c5cc22 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:37.453157 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dacd9699-89c5-4212-883c-5f62b4c5cc22 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:37.453627 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dacd9699-89c5-4212-883c-5f62b4c5cc22 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:37.473082 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-dacd9699-89c5-4212-883c-5f62b4c5cc22 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:43:37.474620 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-dacd9699-89c5-4212-883c-5f62b4c5cc22 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:43:37.483875 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:37.554542 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 430/875] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:43:37 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 09 00:43:37.567263 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2478370-5c2e-4c12-92ba-82724a6cfe42 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:37.603562 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d95d6571-ed2b-4f00-ab0c-f897285e10c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:37.604290 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d95d6571-ed2b-4f00-ab0c-f897285e10c3 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:37.604580 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d95d6571-ed2b-4f00-ab0c-f897285e10c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:37.604854 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d95d6571-ed2b-4f00-ab0c-f897285e10c3 None admin] PATH_INFO: `/v3/services` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:37.605485 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d95d6571-ed2b-4f00-ab0c-f897285e10c3 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:37.605773 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d95d6571-ed2b-4f00-ab0c-f897285e10c3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:37.607141 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d95d6571-ed2b-4f00-ab0c-f897285e10c3 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:37.612335 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 446/876] 10.222.0.33 () {56 vars in 1122 bytes} [Sun Oct 9 00:43:37 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:43:37.618411 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dacd9699-89c5-4212-883c-5f62b4c5cc22 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:37.714199 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:37.728398 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dacd9699-89c5-4212-883c-5f62b4c5cc22 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:37.764261 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0d258c6-1b2b-45d7-8190-53f1dc6f14dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:37.764871 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0d258c6-1b2b-45d7-8190-53f1dc6f14dc None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:37.765156 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0d258c6-1b2b-45d7-8190-53f1dc6f14dc None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:37.765428 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0d258c6-1b2b-45d7-8190-53f1dc6f14dc None admin] PATH_INFO: `/v3/services` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:37.766057 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0d258c6-1b2b-45d7-8190-53f1dc6f14dc None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:37.766361 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0d258c6-1b2b-45d7-8190-53f1dc6f14dc None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:37.767747 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0d258c6-1b2b-45d7-8190-53f1dc6f14dc None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:37.780052 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 431/877] 10.222.0.33 () {56 vars in 1122 bytes} [Sun Oct 9 00:43:37 2022] GET /identity/v3/services?type=object-store => generated 129 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:37.830471 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d95d6571-ed2b-4f00-ab0c-f897285e10c3 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:37.946215 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:37.949212 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d95d6571-ed2b-4f00-ab0c-f897285e10c3 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:37.986468 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a38ef393-0974-4f1b-8fff-c369d4fb175d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:37.987071 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a38ef393-0974-4f1b-8fff-c369d4fb175d None admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:37.987352 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a38ef393-0974-4f1b-8fff-c369d4fb175d None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:37.987649 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a38ef393-0974-4f1b-8fff-c369d4fb175d None admin] PATH_INFO: `/v3/services` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:37.988313 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a38ef393-0974-4f1b-8fff-c369d4fb175d None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:37.989686 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a38ef393-0974-4f1b-8fff-c369d4fb175d None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:38.006388 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 447/878] 10.222.0.33 () {60 vars in 1141 bytes} [Sun Oct 9 00:43:37 2022] POST /identity/v3/services => generated 251 bytes in 176 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:43:38.246000 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b9a24c-e1b0-4d02-9ed9-c8b62e9356a4 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:38.246519 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b9a24c-e1b0-4d02-9ed9-c8b62e9356a4 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:38.246519 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b9a24c-e1b0-4d02-9ed9-c8b62e9356a4 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:38.248241 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 432/879] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:43:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:43:38.253682 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58478ba-0b5f-45a0-a2f7-c4ebcf5aac6a None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:38.253973 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58478ba-0b5f-45a0-a2f7-c4ebcf5aac6a None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:38.254245 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58478ba-0b5f-45a0-a2f7-c4ebcf5aac6a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:38.273456 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-d58478ba-0b5f-45a0-a2f7-c4ebcf5aac6a None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:43:38.274642 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-d58478ba-0b5f-45a0-a2f7-c4ebcf5aac6a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:43:38.351637 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 448/880] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:43:38 2022] POST /identity/v3/auth/tokens => generated 1262 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 09 00:43:38.622689 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36b9a24c-e1b0-4d02-9ed9-c8b62e9356a4 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:38.733190 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36b9a24c-e1b0-4d02-9ed9-c8b62e9356a4 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:38.768772 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-470f8a75-65bc-47ec-bae2-95a6e33bc04b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:38.769388 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-470f8a75-65bc-47ec-bae2-95a6e33bc04b None admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:38.769675 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-470f8a75-65bc-47ec-bae2-95a6e33bc04b None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:38.769951 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-470f8a75-65bc-47ec-bae2-95a6e33bc04b None admin] PATH_INFO: `/v3/services` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:38.770625 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-470f8a75-65bc-47ec-bae2-95a6e33bc04b None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:38.772022 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-470f8a75-65bc-47ec-bae2-95a6e33bc04b None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:38.792264 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 433/881] 10.222.0.33 () {60 vars in 1141 bytes} [Sun Oct 9 00:43:38 2022] POST /identity/v3/services => generated 269 bytes in 170 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:43:39.156636 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0799d98c-f249-4976-b078-356808956f5e None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:39.156977 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0799d98c-f249-4976-b078-356808956f5e None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:39.157179 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0799d98c-f249-4976-b078-356808956f5e None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:39.158831 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 449/882] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:43:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:43:39.164485 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b1cbdd-a9fd-4d4d-b78b-b5bd14c90d36 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:39.164772 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b1cbdd-a9fd-4d4d-b78b-b5bd14c90d36 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:39.165047 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b1cbdd-a9fd-4d4d-b78b-b5bd14c90d36 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:39.188109 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-d7b1cbdd-a9fd-4d4d-b78b-b5bd14c90d36 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:43:39.189314 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-d7b1cbdd-a9fd-4d4d-b78b-b5bd14c90d36 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:43:39.199638 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:39.268516 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 434/883] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:43:39 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 09 00:43:39.535537 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0799d98c-f249-4976-b078-356808956f5e None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:39.626419 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:39.643550 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0799d98c-f249-4976-b078-356808956f5e None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:39.681148 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e417666-c634-4633-b962-ec01d5cd1aeb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:39.682153 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e417666-c634-4633-b962-ec01d5cd1aeb None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:39.682659 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e417666-c634-4633-b962-ec01d5cd1aeb None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:39.683140 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e417666-c634-4633-b962-ec01d5cd1aeb None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:39.693928 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e417666-c634-4633-b962-ec01d5cd1aeb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:43:39.699282 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-6e417666-c634-4633-b962-ec01d5cd1aeb None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 09 00:43:39.699282 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:39.699282 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:39.699282 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:39.699282 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:39.699282 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:39.699282 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:39.699282 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:39.699282 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:39.699282 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:39.699282 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:39.699282 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:39.699282 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:43:39.699282 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:43:39.699282 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:43:39.699282 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:43:39.700285 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:43:39.700285 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:43:39.700285 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:43:39.700285 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:43:39.700285 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:39.700285 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:39.700285 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:39.700285 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:39.700285 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:39.700285 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:39.700285 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:39.700285 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:39.700285 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:39.700285 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:39.700285 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:39.700285 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:39.701158 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:39.701158 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:39.701158 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:39.701158 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:39.701158 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:43:39.701158 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:43:39.701158 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:43:39.701158 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:43:39.701158 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:43:39.701158 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:43:39.701158 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 09 00:43:39.701158 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:39.703798 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 450/884] 10.222.0.33 () {56 vars in 1120 bytes} [Sun Oct 9 00:43:39 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:39.715628 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7b1cbdd-a9fd-4d4d-b78b-b5bd14c90d36 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:39.815819 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7b1cbdd-a9fd-4d4d-b78b-b5bd14c90d36 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:39.851389 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39b4cb93-92e7-478f-94b2-7ca6b61432b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:39.852024 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39b4cb93-92e7-478f-94b2-7ca6b61432b9 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:39.852302 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39b4cb93-92e7-478f-94b2-7ca6b61432b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:39.852574 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39b4cb93-92e7-478f-94b2-7ca6b61432b9 None admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:39.853232 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39b4cb93-92e7-478f-94b2-7ca6b61432b9 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:39.853530 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39b4cb93-92e7-478f-94b2-7ca6b61432b9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:39.854888 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39b4cb93-92e7-478f-94b2-7ca6b61432b9 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:39.863765 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 435/885] 10.222.0.33 () {56 vars in 1115 bytes} [Sun Oct 9 00:43:39 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:39.869210 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e417666-c634-4633-b962-ec01d5cd1aeb None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:39.979659 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e417666-c634-4633-b962-ec01d5cd1aeb None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:40.014476 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e884d917-8c66-492e-9413-af5ac7e6fafa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:40.015100 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e884d917-8c66-492e-9413-af5ac7e6fafa None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:40.015372 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e884d917-8c66-492e-9413-af5ac7e6fafa None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:40.015676 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e884d917-8c66-492e-9413-af5ac7e6fafa None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:40.027955 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e884d917-8c66-492e-9413-af5ac7e6fafa None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:40.029970 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e884d917-8c66-492e-9413-af5ac7e6fafa None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:40.038725 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:40.046917 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-e884d917-8c66-492e-9413-af5ac7e6fafa None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:40.046917 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:40.046917 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:40.046917 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:40.046917 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:40.046917 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:40.046917 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:40.046917 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:40.046917 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:40.046917 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:40.046917 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:40.046917 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:40.046917 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:43:40.046917 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:43:40.046917 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:43:40.046917 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:43:40.048915 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:40.048915 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:40.048915 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:40.048915 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:40.048915 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:40.048915 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:40.048915 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:40.048915 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:40.048915 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:40.048915 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:40.048915 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:40.048915 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:40.048915 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:40.048915 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:40.048915 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:40.048915 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:40.048915 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:43:40.050113 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:43:40.050113 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:40.050113 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:40.050113 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 451/886] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:43:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:40.054380 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39b4cb93-92e7-478f-94b2-7ca6b61432b9 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:40.059246 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:40.061655 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556103a3-56f3-49a2-801e-9f6eae908166 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:40.061934 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556103a3-56f3-49a2-801e-9f6eae908166 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:40.062262 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556103a3-56f3-49a2-801e-9f6eae908166 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:40.063880 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 452/887] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:43:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 09 00:43:40.068889 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f23c346d-5a1a-4944-a9d3-dd7492704d46 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:40.069986 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f23c346d-5a1a-4944-a9d3-dd7492704d46 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:40.070261 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f23c346d-5a1a-4944-a9d3-dd7492704d46 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:40.098298 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-f23c346d-5a1a-4944-a9d3-dd7492704d46 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:43:40.099585 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-f23c346d-5a1a-4944-a9d3-dd7492704d46 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:43:40.172918 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39b4cb93-92e7-478f-94b2-7ca6b61432b9 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:40.187986 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 453/888] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:43:40 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 09 00:43:40.214262 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a596413d-e369-4696-85a4-7d3d4f08efe6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:40.215158 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a596413d-e369-4696-85a4-7d3d4f08efe6 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:40.215628 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a596413d-e369-4696-85a4-7d3d4f08efe6 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:40.216071 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a596413d-e369-4696-85a4-7d3d4f08efe6 None admin] PATH_INFO: `/v3/domains` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:40.216887 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a596413d-e369-4696-85a4-7d3d4f08efe6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:40.217340 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a596413d-e369-4696-85a4-7d3d4f08efe6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:40.218838 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a596413d-e369-4696-85a4-7d3d4f08efe6 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:40.232350 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 436/889] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:43:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:40.242104 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ae9f248-db2c-41d5-b564-525732c08eef None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:40.242104 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ae9f248-db2c-41d5-b564-525732c08eef None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:40.242104 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ae9f248-db2c-41d5-b564-525732c08eef None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:40.242453 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a596413d-e369-4696-85a4-7d3d4f08efe6 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:40.242544 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 454/890] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:43:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:43:40.244655 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f3c2d41-bba9-44ab-8a17-0671bd7607ac None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:40.244942 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f3c2d41-bba9-44ab-8a17-0671bd7607ac None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:40.245220 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f3c2d41-bba9-44ab-8a17-0671bd7607ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:40.265226 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-6f3c2d41-bba9-44ab-8a17-0671bd7607ac None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:43:40.266599 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-6f3c2d41-bba9-44ab-8a17-0671bd7607ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:43:40.304169 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:40.338906 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 455/891] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:43:40 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 09 00:43:40.348637 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a596413d-e369-4696-85a4-7d3d4f08efe6 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:40.385161 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7aa7f36-fec4-4a1e-81da-8439ad7d0457 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:40.385788 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7aa7f36-fec4-4a1e-81da-8439ad7d0457 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:40.386067 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7aa7f36-fec4-4a1e-81da-8439ad7d0457 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:40.386337 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7aa7f36-fec4-4a1e-81da-8439ad7d0457 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:40.394261 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7aa7f36-fec4-4a1e-81da-8439ad7d0457 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:40.395835 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7aa7f36-fec4-4a1e-81da-8439ad7d0457 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:40.407454 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-d7aa7f36-fec4-4a1e-81da-8439ad7d0457 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 09 00:43:40.407454 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:40.407454 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:40.407454 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:40.407454 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:40.407454 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:40.407454 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:40.407454 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:40.407454 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:40.407454 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:40.407454 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:40.407454 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:40.407454 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 09 00:43:40.407454 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 09 00:43:40.407454 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 09 00:43:40.407454 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 09 00:43:40.408444 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:40.408444 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:40.408444 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 09 00:43:40.408444 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 09 00:43:40.408444 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 09 00:43:40.408444 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 09 00:43:40.408444 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:40.408444 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:40.408444 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:40.408444 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:40.408444 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:40.408444 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:40.408444 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:40.408444 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:40.408444 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:40.408444 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:40.408444 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:40.408444 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:40.409979 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:40.409979 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:40.409979 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 09 00:43:40.409979 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 09 00:43:40.409979 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 09 00:43:40.409979 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 09 00:43:40.409979 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 09 00:43:40.409979 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 09 00:43:40.409979 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 09 00:43:40.409979 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:40.409979 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 437/892] 10.222.0.33 () {56 vars in 1117 bytes} [Sun Oct 9 00:43:40 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:40.415294 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f3c2d41-bba9-44ab-8a17-0671bd7607ac None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:40.458928 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7aa7f36-fec4-4a1e-81da-8439ad7d0457 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:40.477565 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:40.534425 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f3c2d41-bba9-44ab-8a17-0671bd7607ac None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:40.574866 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7aa7f36-fec4-4a1e-81da-8439ad7d0457 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:40.576236 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-978cc1e2-a517-49b3-8665-1e076b5e8202 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:40.576839 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-978cc1e2-a517-49b3-8665-1e076b5e8202 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:40.577116 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-978cc1e2-a517-49b3-8665-1e076b5e8202 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:40.577384 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-978cc1e2-a517-49b3-8665-1e076b5e8202 None admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:40.578416 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-978cc1e2-a517-49b3-8665-1e076b5e8202 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:40.578714 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-978cc1e2-a517-49b3-8665-1e076b5e8202 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:40.580765 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-978cc1e2-a517-49b3-8665-1e076b5e8202 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:40.604084 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:40.607275 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 456/893] 10.222.0.33 () {56 vars in 1149 bytes} [Sun Oct 9 00:43:40 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:40.609315 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-978cc1e2-a517-49b3-8665-1e076b5e8202 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:40.617300 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7287c0ca-859a-4f41-85b8-192c6244e378 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:40.617909 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7287c0ca-859a-4f41-85b8-192c6244e378 None admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:40.618189 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7287c0ca-859a-4f41-85b8-192c6244e378 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:40.618463 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7287c0ca-859a-4f41-85b8-192c6244e378 None admin] PATH_INFO: `/v3/services` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:40.619121 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7287c0ca-859a-4f41-85b8-192c6244e378 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:40.620515 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7287c0ca-859a-4f41-85b8-192c6244e378 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:40.636048 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 438/894] 10.222.0.33 () {60 vars in 1141 bytes} [Sun Oct 9 00:43:40 2022] POST /identity/v3/services => generated 241 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 09 00:43:40.636855 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7287c0ca-859a-4f41-85b8-192c6244e378 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:40.748497 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-978cc1e2-a517-49b3-8665-1e076b5e8202 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:40.768776 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:40.781206 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7287c0ca-859a-4f41-85b8-192c6244e378 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:40.792920 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c149590-bfdc-4996-b038-0e048690b42e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:40.793978 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c149590-bfdc-4996-b038-0e048690b42e None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:40.794551 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c149590-bfdc-4996-b038-0e048690b42e None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:40.795034 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c149590-bfdc-4996-b038-0e048690b42e None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:40.795910 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c149590-bfdc-4996-b038-0e048690b42e None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:40.797552 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c149590-bfdc-4996-b038-0e048690b42e None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:40.807319 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-6c149590-bfdc-4996-b038-0e048690b42e None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 09 00:43:40.807319 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:40.807319 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 09 00:43:40.807319 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 09 00:43:40.807319 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 09 00:43:40.807319 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 09 00:43:40.807319 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 09 00:43:40.807319 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 09 00:43:40.807319 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 09 00:43:40.807319 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:40.807319 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 09 00:43:40.807319 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:40.807319 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:40.807319 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:40.807319 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:40.807319 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:40.807319 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:40.808613 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:40.808613 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:40.808613 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:40.808613 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:40.808613 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:40.808613 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:40.808613 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 09 00:43:40.808613 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 09 00:43:40.808613 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 09 00:43:40.808613 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 09 00:43:40.808613 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:40.808613 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:40.808613 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:40.808613 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:40.808613 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:40.808613 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:40.809433 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:40.809433 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:40.809433 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:40.809433 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:40.809433 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:40.809433 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:40.809433 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:40.809433 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:40.809433 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:40.809433 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:40.809433 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 09 00:43:40.809433 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 09 00:43:40.809433 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 09 00:43:40.809433 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:40.812293 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 457/895] 10.222.0.33 () {56 vars in 1114 bytes} [Sun Oct 9 00:43:40 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:40.817808 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c149590-bfdc-4996-b038-0e048690b42e None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:40.829145 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d41b0ac-69ef-4788-9b06-af4256832f28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:40.829766 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d41b0ac-69ef-4788-9b06-af4256832f28 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:40.830048 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d41b0ac-69ef-4788-9b06-af4256832f28 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:40.830324 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d41b0ac-69ef-4788-9b06-af4256832f28 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:40.890164 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:40.917531 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d41b0ac-69ef-4788-9b06-af4256832f28 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:40.919298 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d41b0ac-69ef-4788-9b06-af4256832f28 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:40.944752 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c149590-bfdc-4996-b038-0e048690b42e None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:40.994108 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54ee8131-db41-49eb-a3b3-648bc0c75e7f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:40.994853 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ee8131-db41-49eb-a3b3-648bc0c75e7f None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:40.995200 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ee8131-db41-49eb-a3b3-648bc0c75e7f None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:40.995541 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ee8131-db41-49eb-a3b3-648bc0c75e7f None admin] PATH_INFO: `/v3/services` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:40.996287 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54ee8131-db41-49eb-a3b3-648bc0c75e7f None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:40.996659 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54ee8131-db41-49eb-a3b3-648bc0c75e7f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:40.998101 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54ee8131-db41-49eb-a3b3-648bc0c75e7f None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:41.005148 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 458/896] 10.222.0.33 () {56 vars in 1114 bytes} [Sun Oct 9 00:43:40 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:41.011298 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54ee8131-db41-49eb-a3b3-648bc0c75e7f None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:41.027958 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 439/897] 10.222.0.33 () {58 vars in 1282 bytes} [Sun Oct 9 00:43:40 2022] GET /identity/v3/auth/tokens => generated 1474 bytes in 391 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 09 00:43:41.034704 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d41b0ac-69ef-4788-9b06-af4256832f28 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:41.084596 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:41.131230 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:41.158306 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54ee8131-db41-49eb-a3b3-648bc0c75e7f None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:41.159387 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d41b0ac-69ef-4788-9b06-af4256832f28 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:41.194940 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16c1d178-7c09-4db2-846f-9c3d38555680 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:41.195835 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c1d178-7c09-4db2-846f-9c3d38555680 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:41.196278 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c1d178-7c09-4db2-846f-9c3d38555680 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:41.196701 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c1d178-7c09-4db2-846f-9c3d38555680 None admin] PATH_INFO: `/v3/services` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:41.197465 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16c1d178-7c09-4db2-846f-9c3d38555680 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:41.197681 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d55fb208-5a6e-451b-a01e-2e4daa163dd6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:41.198105 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16c1d178-7c09-4db2-846f-9c3d38555680 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:41.198301 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d55fb208-5a6e-451b-a01e-2e4daa163dd6 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:41.198579 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d55fb208-5a6e-451b-a01e-2e4daa163dd6 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:41.198846 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d55fb208-5a6e-451b-a01e-2e4daa163dd6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:41.200201 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16c1d178-7c09-4db2-846f-9c3d38555680 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:41.206096 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 459/898] 10.222.0.33 () {56 vars in 1114 bytes} [Sun Oct 9 00:43:41 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:43:41.207617 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc741494-5f24-4b77-8ec6-16606b30b721 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:41.207905 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc741494-5f24-4b77-8ec6-16606b30b721 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:41.208175 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc741494-5f24-4b77-8ec6-16606b30b721 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:41.209498 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 460/899] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:43:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:43:41.211894 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc741494-5f24-4b77-8ec6-16606b30b721 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:41.214345 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d55fb208-5a6e-451b-a01e-2e4daa163dd6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:41.216366 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d55fb208-5a6e-451b-a01e-2e4daa163dd6 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:41.235110 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-d55fb208-5a6e-451b-a01e-2e4daa163dd6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:41.235110 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:41.235110 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:41.235110 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:41.235110 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:41.235110 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:41.235110 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:41.235110 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:41.235110 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:41.235110 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:41.235110 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:41.235110 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:41.235110 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:43:41.235110 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:43:41.235110 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:43:41.235110 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:43:41.236113 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:41.236113 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:41.236113 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:41.236113 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:41.236113 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:41.236113 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:41.236113 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:41.236113 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:41.236113 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:41.236113 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:41.236113 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:41.236113 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:41.236113 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:41.236113 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:41.236113 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:41.236113 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:41.236113 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:43:41.237002 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:43:41.237002 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:41.237002 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:41.237002 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 440/900] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:43:41 2022] GET /identity/v3/domains/Default => generated 87 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:41.238067 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85b64254-412a-43e9-941f-d048234631f8 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:41.238067 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85b64254-412a-43e9-941f-d048234631f8 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:41.238254 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85b64254-412a-43e9-941f-d048234631f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:41.257418 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-85b64254-412a-43e9-941f-d048234631f8 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:43:41.258632 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-85b64254-412a-43e9-941f-d048234631f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:43:41.320218 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc741494-5f24-4b77-8ec6-16606b30b721 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:41.328886 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:41.331314 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 441/901] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:43:41 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 09 00:43:41.332340 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85b64254-412a-43e9-941f-d048234631f8 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:41.362252 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dafae7ea-6a6c-4c53-97d6-6499ac3a3a09 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:41.363282 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dafae7ea-6a6c-4c53-97d6-6499ac3a3a09 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:41.363724 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dafae7ea-6a6c-4c53-97d6-6499ac3a3a09 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:41.364128 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dafae7ea-6a6c-4c53-97d6-6499ac3a3a09 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:41.365078 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dafae7ea-6a6c-4c53-97d6-6499ac3a3a09 None admin] RBAC: Adding query filter params (interface=public, service_id=7bf5ebf857af460fb1cecd7e36effd65, region_id=RegionOne) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:41.365443 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dafae7ea-6a6c-4c53-97d6-6499ac3a3a09 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:41.367559 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dafae7ea-6a6c-4c53-97d6-6499ac3a3a09 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:41.379754 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 461/902] 10.222.0.33 () {56 vars in 1251 bytes} [Sun Oct 9 00:43:41 2022] GET /identity/v3/endpoints?service_id=7bf5ebf857af460fb1cecd7e36effd65&interface=public®ion_id=RegionOne => generated 194 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:43:41.385180 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dafae7ea-6a6c-4c53-97d6-6499ac3a3a09 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:41.402139 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:41.451341 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85b64254-412a-43e9-941f-d048234631f8 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:41.489163 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-329a710c-5ce2-468f-9b6e-da9e8b4c50f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:41.489775 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-329a710c-5ce2-468f-9b6e-da9e8b4c50f9 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:41.490055 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-329a710c-5ce2-468f-9b6e-da9e8b4c50f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:41.490328 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-329a710c-5ce2-468f-9b6e-da9e8b4c50f9 None admin] PATH_INFO: `/v3/domains` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:41.490992 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-329a710c-5ce2-468f-9b6e-da9e8b4c50f9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:41.491333 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-329a710c-5ce2-468f-9b6e-da9e8b4c50f9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:41.492879 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-329a710c-5ce2-468f-9b6e-da9e8b4c50f9 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:41.504558 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 442/903] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:43:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:41.510280 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-329a710c-5ce2-468f-9b6e-da9e8b4c50f9 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:41.524610 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dafae7ea-6a6c-4c53-97d6-6499ac3a3a09 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:41.564477 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f23f43a-b22c-4ac3-b64f-097e6a38f8fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:41.564816 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f23f43a-b22c-4ac3-b64f-097e6a38f8fb None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:41.565130 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f23f43a-b22c-4ac3-b64f-097e6a38f8fb None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:41.565395 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f23f43a-b22c-4ac3-b64f-097e6a38f8fb None admin] PATH_INFO: `/v3/services` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:41.565992 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f23f43a-b22c-4ac3-b64f-097e6a38f8fb None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:41.567661 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f23f43a-b22c-4ac3-b64f-097e6a38f8fb None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:41.580350 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 462/904] 10.222.0.33 () {56 vars in 1087 bytes} [Sun Oct 9 00:43:41 2022] GET /identity/v3/services => generated 1497 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 09 00:43:41.598890 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f23f43a-b22c-4ac3-b64f-097e6a38f8fb None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:41.643332 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:41.652070 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-329a710c-5ce2-468f-9b6e-da9e8b4c50f9 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:41.682123 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:41.701329 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-effa02d3-98eb-4f10-a4c6-b6cdd25a7d88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:41.702086 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-effa02d3-98eb-4f10-a4c6-b6cdd25a7d88 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:41.702313 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-effa02d3-98eb-4f10-a4c6-b6cdd25a7d88 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:41.702587 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-effa02d3-98eb-4f10-a4c6-b6cdd25a7d88 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:41.712289 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-effa02d3-98eb-4f10-a4c6-b6cdd25a7d88 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:41.714140 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-effa02d3-98eb-4f10-a4c6-b6cdd25a7d88 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:41.725018 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f23f43a-b22c-4ac3-b64f-097e6a38f8fb None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:41.732565 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-effa02d3-98eb-4f10-a4c6-b6cdd25a7d88 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 09 00:43:41.732565 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:41.732565 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:41.732565 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:41.732565 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:41.732565 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:41.732565 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:41.732565 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:41.732565 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:41.732565 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:41.732565 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:41.732565 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:41.732565 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 09 00:43:41.732565 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 09 00:43:41.732565 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 09 00:43:41.733522 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 09 00:43:41.733522 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:41.733522 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:41.733522 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:41.733522 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:41.733522 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:41.733522 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:41.733522 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:41.733522 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:41.733522 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:41.733522 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:41.733522 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:41.733522 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:41.733522 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:41.733522 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:41.733522 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:41.733522 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:41.734398 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 09 00:43:41.734398 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 09 00:43:41.734398 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 09 00:43:41.734398 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 09 00:43:41.734398 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 09 00:43:41.734398 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 09 00:43:41.734398 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 09 00:43:41.734398 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:41.734842 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 443/905] 10.222.0.33 () {56 vars in 1111 bytes} [Sun Oct 9 00:43:41 2022] GET /identity/v3/projects/service => generated 88 bytes in 224 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 09 00:43:41.740069 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-effa02d3-98eb-4f10-a4c6-b6cdd25a7d88 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:41.776622 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-961d24c6-8e0c-41d2-aa32-099747d7c583 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:41.777406 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961d24c6-8e0c-41d2-aa32-099747d7c583 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:41.777698 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961d24c6-8e0c-41d2-aa32-099747d7c583 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:41.778268 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961d24c6-8e0c-41d2-aa32-099747d7c583 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:41.779124 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-961d24c6-8e0c-41d2-aa32-099747d7c583 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:41.780904 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-961d24c6-8e0c-41d2-aa32-099747d7c583 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:41.793477 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-961d24c6-8e0c-41d2-aa32-099747d7c583 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 09 00:43:41.793477 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:41.793477 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 09 00:43:41.793477 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 09 00:43:41.793477 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 09 00:43:41.793477 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 09 00:43:41.793477 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 09 00:43:41.793477 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 09 00:43:41.793477 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 09 00:43:41.793477 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:41.793477 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 09 00:43:41.793477 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:41.793477 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:41.793477 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:41.793477 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:41.793477 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:41.794487 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:41.794487 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:41.794487 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:41.794487 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:41.794487 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:41.794487 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:41.794487 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:41.794487 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 09 00:43:41.794487 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 09 00:43:41.794487 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 09 00:43:41.794487 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 09 00:43:41.794487 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:41.794487 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:41.794487 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:41.794487 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:41.794487 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:41.794487 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:41.795526 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:41.795526 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:41.795526 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:41.795526 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:41.795526 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:41.795526 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:41.795526 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:41.795526 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:41.795526 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:41.795526 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:41.795526 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 09 00:43:41.795526 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 09 00:43:41.795526 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 09 00:43:41.795526 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:41.795526 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 463/906] 10.222.0.33 () {56 vars in 1132 bytes} [Sun Oct 9 00:43:41 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 09 00:43:41.801133 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-961d24c6-8e0c-41d2-aa32-099747d7c583 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:41.867671 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-effa02d3-98eb-4f10-a4c6-b6cdd25a7d88 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:41.906328 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3e47152-d604-43c8-b907-ab086d549fad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:41.906934 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3e47152-d604-43c8-b907-ab086d549fad None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:41.907211 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3e47152-d604-43c8-b907-ab086d549fad None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:41.907478 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3e47152-d604-43c8-b907-ab086d549fad None admin] PATH_INFO: `/v3/projects` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:41.908288 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3e47152-d604-43c8-b907-ab086d549fad None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:41.908593 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3e47152-d604-43c8-b907-ab086d549fad None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:41.910000 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3e47152-d604-43c8-b907-ab086d549fad None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:41.912226 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:41.917305 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-961d24c6-8e0c-41d2-aa32-099747d7c583 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:41.931617 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 444/907] 10.222.0.33 () {56 vars in 1148 bytes} [Sun Oct 9 00:43:41 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:43:41.933497 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3e47152-d604-43c8-b907-ab086d549fad None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:41.960224 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb4973b0-3da9-475b-8797-e7f51cdf75d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:41.960907 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb4973b0-3da9-475b-8797-e7f51cdf75d9 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:41.961037 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb4973b0-3da9-475b-8797-e7f51cdf75d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:41.961274 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb4973b0-3da9-475b-8797-e7f51cdf75d9 None admin] PATH_INFO: `/v3/services` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:41.961945 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb4973b0-3da9-475b-8797-e7f51cdf75d9 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:41.962319 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb4973b0-3da9-475b-8797-e7f51cdf75d9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:41.963716 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb4973b0-3da9-475b-8797-e7f51cdf75d9 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:41.969104 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 464/908] 10.222.0.33 () {56 vars in 1126 bytes} [Sun Oct 9 00:43:41 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:41.974547 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb4973b0-3da9-475b-8797-e7f51cdf75d9 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:41.980920 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:42.049140 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3e47152-d604-43c8-b907-ab086d549fad None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:42.088512 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b752527c-4e37-4cf2-a224-59935d2ad9fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:42.089387 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b752527c-4e37-4cf2-a224-59935d2ad9fe None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:42.089708 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb4973b0-3da9-475b-8797-e7f51cdf75d9 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:42.090090 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b752527c-4e37-4cf2-a224-59935d2ad9fe None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:42.090594 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b752527c-4e37-4cf2-a224-59935d2ad9fe None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:42.091510 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b752527c-4e37-4cf2-a224-59935d2ad9fe None admin] RBAC: Adding query filter params (role.id=fddb7718710e48bb8034bd9390b820a2, scope.project.id=013c83476b1340fbb051061833c4a9c6, user.id=5f3247ddcda54394995d844c776bf43c) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:42.092084 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b752527c-4e37-4cf2-a224-59935d2ad9fe None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:42.093724 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b752527c-4e37-4cf2-a224-59935d2ad9fe None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:42.100075 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 445/909] 10.222.0.33 () {56 vars in 1374 bytes} [Sun Oct 9 00:43:41 2022] GET /identity/v3/role_assignments?user.id=5f3247ddcda54394995d844c776bf43c&scope.project.id=013c83476b1340fbb051061833c4a9c6&role.id=fddb7718710e48bb8034bd9390b820a2 => generated 259 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:43:42.136610 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2f3c4d8-9212-410c-a24f-32e722f5ad43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:42.137632 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2f3c4d8-9212-410c-a24f-32e722f5ad43 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:42.138131 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2f3c4d8-9212-410c-a24f-32e722f5ad43 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:42.138683 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2f3c4d8-9212-410c-a24f-32e722f5ad43 None admin] PATH_INFO: `/v3/services` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:42.139560 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2f3c4d8-9212-410c-a24f-32e722f5ad43 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:42.140202 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2f3c4d8-9212-410c-a24f-32e722f5ad43 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:42.141845 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2f3c4d8-9212-410c-a24f-32e722f5ad43 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:42.148317 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 465/910] 10.222.0.33 () {56 vars in 1126 bytes} [Sun Oct 9 00:43:41 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:43:42.156655 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b752527c-4e37-4cf2-a224-59935d2ad9fe None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:42.271382 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b752527c-4e37-4cf2-a224-59935d2ad9fe None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:42.273787 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:42.314505 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f3c8af9-f80b-417c-adce-8db419f70bed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:42.315116 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f3c8af9-f80b-417c-adce-8db419f70bed None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:42.315392 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f3c8af9-f80b-417c-adce-8db419f70bed None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:42.315693 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f3c8af9-f80b-417c-adce-8db419f70bed None admin] PATH_INFO: `/v3/endpoints` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:42.316385 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f3c8af9-f80b-417c-adce-8db419f70bed None admin] RBAC: Adding query filter params (interface=public, service_id=b89a61ead776420dad1799aef39bb9e0, region_id=RegionOne) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:42.316688 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f3c8af9-f80b-417c-adce-8db419f70bed None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:42.318041 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f3c8af9-f80b-417c-adce-8db419f70bed None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:42.324400 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 446/911] 10.222.0.33 () {56 vars in 1251 bytes} [Sun Oct 9 00:43:42 2022] GET /identity/v3/endpoints?service_id=b89a61ead776420dad1799aef39bb9e0&interface=public®ion_id=RegionOne => generated 194 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:42.329992 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2f3c4d8-9212-410c-a24f-32e722f5ad43 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:42.360161 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:42.437928 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2f3c4d8-9212-410c-a24f-32e722f5ad43 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:42.479684 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94ac1188-063f-460a-b7e8-3b74309ad221 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:42.480291 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ac1188-063f-460a-b7e8-3b74309ad221 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:42.480573 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ac1188-063f-460a-b7e8-3b74309ad221 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:42.480843 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ac1188-063f-460a-b7e8-3b74309ad221 None admin] PATH_INFO: `/v3/services` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:42.481442 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94ac1188-063f-460a-b7e8-3b74309ad221 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:42.482874 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94ac1188-063f-460a-b7e8-3b74309ad221 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:42.489376 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 466/912] 10.222.0.33 () {56 vars in 1087 bytes} [Sun Oct 9 00:43:42 2022] GET /identity/v3/services => generated 1497 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:43:42.939844 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9227c529-3c13-4b14-a02f-218cb6e7e7a0 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:42.940395 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9227c529-3c13-4b14-a02f-218cb6e7e7a0 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:42.940395 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9227c529-3c13-4b14-a02f-218cb6e7e7a0 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:42.941827 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 447/913] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:43:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:43:42.947695 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfa8732f-a765-4273-817e-29b579cd9338 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:42.948523 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfa8732f-a765-4273-817e-29b579cd9338 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:42.949006 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfa8732f-a765-4273-817e-29b579cd9338 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:42.968491 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-dfa8732f-a765-4273-817e-29b579cd9338 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:43:42.970017 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-dfa8732f-a765-4273-817e-29b579cd9338 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:43:43.041014 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 467/914] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:43:42 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 09 00:43:43.319685 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9227c529-3c13-4b14-a02f-218cb6e7e7a0 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:43.424262 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9227c529-3c13-4b14-a02f-218cb6e7e7a0 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:43.462223 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d6dacf1-5da4-43f4-84f5-629ac9e1ba51 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:43.462867 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d6dacf1-5da4-43f4-84f5-629ac9e1ba51 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:43.463149 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d6dacf1-5da4-43f4-84f5-629ac9e1ba51 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:43.463422 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d6dacf1-5da4-43f4-84f5-629ac9e1ba51 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:43.464120 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d6dacf1-5da4-43f4-84f5-629ac9e1ba51 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:43.465552 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d6dacf1-5da4-43f4-84f5-629ac9e1ba51 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:43.474791 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-6d6dacf1-5da4-43f4-84f5-629ac9e1ba51 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 09 00:43:43.474791 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:43.474791 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 09 00:43:43.474791 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 09 00:43:43.474791 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 09 00:43:43.474791 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 09 00:43:43.474791 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 09 00:43:43.474791 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 09 00:43:43.474791 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 09 00:43:43.474791 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:43.474791 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 09 00:43:43.474791 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:43.474791 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:43.474791 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:43.474791 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:43.474791 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:43.474791 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:43.476366 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:43.476366 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:43.476366 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:43.476366 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:43.476366 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:43.476366 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:43.476366 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 09 00:43:43.476366 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 09 00:43:43.476366 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 09 00:43:43.476366 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 09 00:43:43.476366 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:43.476366 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:43.476366 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:43.476366 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:43.476366 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:43.476366 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:43.477184 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:43.477184 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:43.477184 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:43.477184 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:43.477184 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:43.477184 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:43.477184 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:43.477184 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:43.477184 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:43.477184 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:43.477184 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 09 00:43:43.477184 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 09 00:43:43.477184 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 09 00:43:43.477184 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:43.477184 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 448/915] 10.222.0.33 () {56 vars in 1126 bytes} [Sun Oct 9 00:43:43 2022] GET /identity/v3/services/object-store => generated 93 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:43.484526 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfa8732f-a765-4273-817e-29b579cd9338 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:43.515971 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:43.586668 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfa8732f-a765-4273-817e-29b579cd9338 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:43.622546 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d039f2d-957e-4106-b107-524ac229358f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:43.623497 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d039f2d-957e-4106-b107-524ac229358f None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:43.624023 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d039f2d-957e-4106-b107-524ac229358f None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:43.624488 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d039f2d-957e-4106-b107-524ac229358f None admin] PATH_INFO: `/v3/services` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:43.625321 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d039f2d-957e-4106-b107-524ac229358f None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:43.625831 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d039f2d-957e-4106-b107-524ac229358f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:43.627380 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d039f2d-957e-4106-b107-524ac229358f None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:43.633333 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 468/916] 10.222.0.33 () {56 vars in 1122 bytes} [Sun Oct 9 00:43:43 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:43.639144 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d6dacf1-5da4-43f4-84f5-629ac9e1ba51 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:43.671480 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:43.745013 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d6dacf1-5da4-43f4-84f5-629ac9e1ba51 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:43.785287 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a2edd81-c10c-4031-873e-542fa2cdca42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:43.785903 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2edd81-c10c-4031-873e-542fa2cdca42 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:43.786182 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2edd81-c10c-4031-873e-542fa2cdca42 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:43.786454 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2edd81-c10c-4031-873e-542fa2cdca42 None admin] PATH_INFO: `/v3/services` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:43.787087 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a2edd81-c10c-4031-873e-542fa2cdca42 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:43.787384 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a2edd81-c10c-4031-873e-542fa2cdca42 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:43.789136 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a2edd81-c10c-4031-873e-542fa2cdca42 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:43.794914 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 449/917] 10.222.0.33 () {56 vars in 1122 bytes} [Sun Oct 9 00:43:43 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:43.800469 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d039f2d-957e-4106-b107-524ac229358f None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:43.889022 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8ef3b5c-5250-4fbe-82cd-55833287bd65 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:43.889324 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8ef3b5c-5250-4fbe-82cd-55833287bd65 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:43.889519 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8ef3b5c-5250-4fbe-82cd-55833287bd65 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:43.889868 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:43.892360 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 450/918] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:43:43 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 09 00:43:43.899625 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c93839b-a033-4cf4-a0de-c65eb4868d00 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:43.899907 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c93839b-a033-4cf4-a0de-c65eb4868d00 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:43.900184 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c93839b-a033-4cf4-a0de-c65eb4868d00 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:43.911283 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d039f2d-957e-4106-b107-524ac229358f None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:43.920808 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-0c93839b-a033-4cf4-a0de-c65eb4868d00 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:43:43.922479 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-0c93839b-a033-4cf4-a0de-c65eb4868d00 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:43:43.948479 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49676b7a-aa35-4366-8b27-1fe37c527553 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:43.949133 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49676b7a-aa35-4366-8b27-1fe37c527553 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:43.949362 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49676b7a-aa35-4366-8b27-1fe37c527553 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:43.949639 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49676b7a-aa35-4366-8b27-1fe37c527553 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:43.950331 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49676b7a-aa35-4366-8b27-1fe37c527553 None admin] RBAC: Adding query filter params (interface=public, service_id=2eb616ab91c04b3bacba0ee7173bf8c1, region_id=RegionOne) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:43.950656 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49676b7a-aa35-4366-8b27-1fe37c527553 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:43.952071 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49676b7a-aa35-4366-8b27-1fe37c527553 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:43.957572 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 469/919] 10.222.0.33 () {56 vars in 1251 bytes} [Sun Oct 9 00:43:43 2022] GET /identity/v3/endpoints?service_id=2eb616ab91c04b3bacba0ee7173bf8c1&interface=public®ion_id=RegionOne => generated 194 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:43.964432 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49676b7a-aa35-4366-8b27-1fe37c527553 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:44.006028 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 451/920] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:43:43 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 09 00:43:44.086313 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49676b7a-aa35-4366-8b27-1fe37c527553 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:44.122776 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3cbb997-ad6b-461c-8e11-8aaf7341b68e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:44.123685 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3cbb997-ad6b-461c-8e11-8aaf7341b68e None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:44.124111 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3cbb997-ad6b-461c-8e11-8aaf7341b68e None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:44.124527 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3cbb997-ad6b-461c-8e11-8aaf7341b68e None admin] PATH_INFO: `/v3/services` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:44.125290 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3cbb997-ad6b-461c-8e11-8aaf7341b68e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:44.126792 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3cbb997-ad6b-461c-8e11-8aaf7341b68e None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:44.133364 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 470/921] 10.222.0.33 () {56 vars in 1087 bytes} [Sun Oct 9 00:43:43 2022] GET /identity/v3/services => generated 1497 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:43:44.306414 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c93839b-a033-4cf4-a0de-c65eb4868d00 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:44.337765 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:44.419028 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c93839b-a033-4cf4-a0de-c65eb4868d00 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:44.455026 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b721322d-2f57-488d-97ac-16c18f21122a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:44.456085 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b721322d-2f57-488d-97ac-16c18f21122a None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:44.456590 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b721322d-2f57-488d-97ac-16c18f21122a None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:44.457089 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b721322d-2f57-488d-97ac-16c18f21122a None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:44.457886 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b721322d-2f57-488d-97ac-16c18f21122a None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:44.459582 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b721322d-2f57-488d-97ac-16c18f21122a None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:44.469678 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-b721322d-2f57-488d-97ac-16c18f21122a None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 09 00:43:44.469678 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:44.469678 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 09 00:43:44.469678 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 09 00:43:44.469678 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 09 00:43:44.469678 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 09 00:43:44.469678 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 09 00:43:44.469678 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 09 00:43:44.469678 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 09 00:43:44.469678 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:44.469678 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 09 00:43:44.469678 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:44.469678 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:44.469678 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:44.469678 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:44.469678 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:44.469678 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:44.470717 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:44.470717 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:44.470717 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:44.470717 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:44.470717 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:44.470717 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:44.470717 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 09 00:43:44.470717 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 09 00:43:44.470717 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 09 00:43:44.470717 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 09 00:43:44.470717 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:44.470717 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:44.470717 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:44.470717 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:44.470717 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:44.470717 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:44.471546 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:44.471546 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:44.471546 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:44.471546 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:44.471546 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:44.471546 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:44.471546 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:44.471546 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:44.471546 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:44.471546 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:44.471546 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 09 00:43:44.471546 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 09 00:43:44.471546 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 09 00:43:44.471546 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:44.477302 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 452/922] 10.222.0.33 () {56 vars in 1114 bytes} [Sun Oct 9 00:43:44 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:44.477766 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d039db0a-c1ff-473c-aee3-0462ac729541 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:44.478211 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d039db0a-c1ff-473c-aee3-0462ac729541 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:44.478618 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d039db0a-c1ff-473c-aee3-0462ac729541 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:44.481089 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 471/923] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:43:44 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 09 00:43:44.482841 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b721322d-2f57-488d-97ac-16c18f21122a None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:44.487337 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-311811b8-f6ba-4fad-9401-0fbe14222167 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:44.487655 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-311811b8-f6ba-4fad-9401-0fbe14222167 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:44.487937 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-311811b8-f6ba-4fad-9401-0fbe14222167 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:44.509684 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-311811b8-f6ba-4fad-9401-0fbe14222167 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:43:44.511602 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-311811b8-f6ba-4fad-9401-0fbe14222167 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:43:44.525761 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:44.590354 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:44.605195 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b721322d-2f57-488d-97ac-16c18f21122a None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:44.612201 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 472/924] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:43:44 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 09 00:43:44.648443 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8df1b622-6e5b-4ecb-b635-23fd0c8dc412 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:44.649109 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8df1b622-6e5b-4ecb-b635-23fd0c8dc412 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:44.649422 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8df1b622-6e5b-4ecb-b635-23fd0c8dc412 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:44.649708 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8df1b622-6e5b-4ecb-b635-23fd0c8dc412 None admin] PATH_INFO: `/v3/services` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:44.650410 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8df1b622-6e5b-4ecb-b635-23fd0c8dc412 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:44.650721 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8df1b622-6e5b-4ecb-b635-23fd0c8dc412 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:44.652156 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8df1b622-6e5b-4ecb-b635-23fd0c8dc412 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:44.662182 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 453/925] 10.222.0.33 () {56 vars in 1114 bytes} [Sun Oct 9 00:43:44 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:44.663894 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-311811b8-f6ba-4fad-9401-0fbe14222167 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:44.717101 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dbb6611-4661-448f-ad5e-449d096001bd None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:44.717429 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dbb6611-4661-448f-ad5e-449d096001bd None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:44.717666 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dbb6611-4661-448f-ad5e-449d096001bd None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:44.719244 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 454/926] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:43:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 09 00:43:44.724330 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-585cf540-9329-451c-bbfc-c9c4cc28a8d7 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:44.724618 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-585cf540-9329-451c-bbfc-c9c4cc28a8d7 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:44.724897 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-585cf540-9329-451c-bbfc-c9c4cc28a8d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:44.744707 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-585cf540-9329-451c-bbfc-c9c4cc28a8d7 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:43:44.745911 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-585cf540-9329-451c-bbfc-c9c4cc28a8d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:43:44.786478 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-311811b8-f6ba-4fad-9401-0fbe14222167 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:44.824738 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 455/927] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:43:44 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 09 00:43:44.830947 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc0782de-c0be-4d5d-aff1-a451a6adb000 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:44.831774 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc0782de-c0be-4d5d-aff1-a451a6adb000 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:44.832217 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc0782de-c0be-4d5d-aff1-a451a6adb000 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:44.832645 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc0782de-c0be-4d5d-aff1-a451a6adb000 None admin] PATH_INFO: `/v3/services` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:44.833417 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc0782de-c0be-4d5d-aff1-a451a6adb000 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:44.833924 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc0782de-c0be-4d5d-aff1-a451a6adb000 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:44.836194 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc0782de-c0be-4d5d-aff1-a451a6adb000 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:44.846102 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 473/928] 10.222.0.33 () {56 vars in 1114 bytes} [Sun Oct 9 00:43:44 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:44.847531 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-585cf540-9329-451c-bbfc-c9c4cc28a8d7 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:44.906843 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc0782de-c0be-4d5d-aff1-a451a6adb000 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:44.918807 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:44.943174 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:44.965756 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-585cf540-9329-451c-bbfc-c9c4cc28a8d7 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:45.014414 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e19c3e37-a600-482b-92c7-c82c5ccb1a38 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:45.016072 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e19c3e37-a600-482b-92c7-c82c5ccb1a38 None admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:45.016796 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e19c3e37-a600-482b-92c7-c82c5ccb1a38 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:45.017265 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e19c3e37-a600-482b-92c7-c82c5ccb1a38 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:45.018802 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e19c3e37-a600-482b-92c7-c82c5ccb1a38 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:45.020379 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e19c3e37-a600-482b-92c7-c82c5ccb1a38 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:45.020638 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc0782de-c0be-4d5d-aff1-a451a6adb000 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:45.051537 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 456/929] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:43:44 2022] POST /identity/v3/endpoints => generated 346 bytes in 204 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 09 00:43:45.072494 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46886f94-60d6-4693-9871-840904d4b213 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:45.073530 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46886f94-60d6-4693-9871-840904d4b213 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:45.074041 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46886f94-60d6-4693-9871-840904d4b213 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:45.074539 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46886f94-60d6-4693-9871-840904d4b213 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:45.095624 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46886f94-60d6-4693-9871-840904d4b213 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:43:45.095624 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-46886f94-60d6-4693-9871-840904d4b213 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 09 00:43:45.095624 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:45.095624 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:45.095624 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:45.095624 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:45.095624 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:45.095624 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:45.095624 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:45.095624 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:45.095624 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:45.095624 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:45.095624 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:45.095624 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:43:45.096741 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:43:45.096741 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:43:45.096741 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:43:45.096741 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:43:45.096741 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:43:45.096741 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:43:45.096741 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:43:45.096741 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:45.096741 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:45.096741 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:45.096741 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:45.096741 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:45.096741 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:45.096741 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:45.096741 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:45.096741 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:45.096741 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:45.097630 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:45.097630 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:45.097630 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:45.097630 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:45.097630 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:45.097630 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:45.097630 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:43:45.097630 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:43:45.097630 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:43:45.097630 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:43:45.097630 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:43:45.097630 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:43:45.097630 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 09 00:43:45.097630 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:45.097630 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 474/930] 10.222.0.33 () {56 vars in 1120 bytes} [Sun Oct 9 00:43:44 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 09 00:43:45.098404 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e19c3e37-a600-482b-92c7-c82c5ccb1a38 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:45.104617 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46886f94-60d6-4693-9871-840904d4b213 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:45.180767 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:45.225278 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46886f94-60d6-4693-9871-840904d4b213 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:45.246173 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e19c3e37-a600-482b-92c7-c82c5ccb1a38 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:45.275278 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99a96df1-4e94-4de7-a0b0-946d6d1221dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:45.275929 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99a96df1-4e94-4de7-a0b0-946d6d1221dc None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:45.276203 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99a96df1-4e94-4de7-a0b0-946d6d1221dc None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:45.276475 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99a96df1-4e94-4de7-a0b0-946d6d1221dc None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:45.277132 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99a96df1-4e94-4de7-a0b0-946d6d1221dc None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:45.277434 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99a96df1-4e94-4de7-a0b0-946d6d1221dc None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:45.278796 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99a96df1-4e94-4de7-a0b0-946d6d1221dc None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:45.285976 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa8a5bb8-ee6b-43ab-ae95-4a3486567895 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:45.286615 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa8a5bb8-ee6b-43ab-ae95-4a3486567895 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:45.286903 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa8a5bb8-ee6b-43ab-ae95-4a3486567895 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:45.287353 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa8a5bb8-ee6b-43ab-ae95-4a3486567895 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:45.287460 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 475/931] 10.222.0.33 () {56 vars in 1115 bytes} [Sun Oct 9 00:43:45 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:45.287898 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa8a5bb8-ee6b-43ab-ae95-4a3486567895 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:45.289599 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa8a5bb8-ee6b-43ab-ae95-4a3486567895 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:45.293192 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99a96df1-4e94-4de7-a0b0-946d6d1221dc None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:45.300089 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-aa8a5bb8-ee6b-43ab-ae95-4a3486567895 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 09 00:43:45.300089 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:45.300089 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 09 00:43:45.300089 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 09 00:43:45.300089 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 09 00:43:45.300089 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 09 00:43:45.300089 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 09 00:43:45.300089 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 09 00:43:45.300089 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 09 00:43:45.300089 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:45.300089 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 09 00:43:45.300089 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:45.300089 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:45.300089 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:45.300089 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:45.300089 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:45.301017 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:45.301017 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:45.301017 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:45.301017 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:45.301017 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:45.301017 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:45.301017 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:45.301017 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 09 00:43:45.301017 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 09 00:43:45.301017 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 09 00:43:45.301017 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 09 00:43:45.301017 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:45.301017 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:45.301017 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:45.301017 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:45.301017 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:45.301017 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:45.301895 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:45.301895 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:45.301895 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:45.301895 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:45.301895 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:45.301895 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:45.301895 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:45.301895 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:45.301895 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:45.301895 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:45.301895 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 09 00:43:45.301895 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 09 00:43:45.301895 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 09 00:43:45.301895 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:45.301895 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 457/932] 10.222.0.33 () {56 vars in 1132 bytes} [Sun Oct 9 00:43:45 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 09 00:43:45.307476 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa8a5bb8-ee6b-43ab-ae95-4a3486567895 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:45.311858 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:45.424853 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99a96df1-4e94-4de7-a0b0-946d6d1221dc None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:45.430961 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa8a5bb8-ee6b-43ab-ae95-4a3486567895 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:45.454331 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:45.464839 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69dfb6b3-1922-4b9e-a421-0232c8503198 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:45.465841 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69dfb6b3-1922-4b9e-a421-0232c8503198 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:45.466399 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69dfb6b3-1922-4b9e-a421-0232c8503198 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:45.466972 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69dfb6b3-1922-4b9e-a421-0232c8503198 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:45.472643 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1709b726-1d21-4ca0-a9c2-2a077a6ed605 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:45.473256 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1709b726-1d21-4ca0-a9c2-2a077a6ed605 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:45.473538 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1709b726-1d21-4ca0-a9c2-2a077a6ed605 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:45.473813 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1709b726-1d21-4ca0-a9c2-2a077a6ed605 None admin] PATH_INFO: `/v3/services` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:45.474443 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1709b726-1d21-4ca0-a9c2-2a077a6ed605 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:45.474743 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1709b726-1d21-4ca0-a9c2-2a077a6ed605 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:45.476668 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1709b726-1d21-4ca0-a9c2-2a077a6ed605 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:45.481899 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69dfb6b3-1922-4b9e-a421-0232c8503198 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:45.483566 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69dfb6b3-1922-4b9e-a421-0232c8503198 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:45.483777 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 458/933] 10.222.0.33 () {56 vars in 1126 bytes} [Sun Oct 9 00:43:45 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:43:45.489962 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1709b726-1d21-4ca0-a9c2-2a077a6ed605 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:45.506192 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-69dfb6b3-1922-4b9e-a421-0232c8503198 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:45.506192 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:45.506192 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:45.506192 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:45.506192 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:45.506192 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:45.506192 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:45.506192 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:45.506192 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:45.506192 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:45.506192 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:45.506192 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:45.506192 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:43:45.506192 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:43:45.506192 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:43:45.506192 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:43:45.507241 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:45.507241 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:45.507241 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:45.507241 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:45.507241 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:45.507241 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:45.507241 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:45.507241 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:45.507241 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:45.507241 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:45.507241 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:45.507241 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:45.507241 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:45.507241 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:45.507241 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:45.507241 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:45.507241 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:43:45.508257 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:43:45.508257 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:45.508257 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:45.512089 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 476/934] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:43:45 2022] GET /identity/v3/domains/Default => generated 87 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 09 00:43:45.513387 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69dfb6b3-1922-4b9e-a421-0232c8503198 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:45.571731 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:45.608756 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1709b726-1d21-4ca0-a9c2-2a077a6ed605 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:45.639426 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69dfb6b3-1922-4b9e-a421-0232c8503198 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:45.654795 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e627dd2-52e5-492c-8987-bf25dd9bc260 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:45.655887 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e627dd2-52e5-492c-8987-bf25dd9bc260 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:45.656319 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e627dd2-52e5-492c-8987-bf25dd9bc260 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:45.656986 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e627dd2-52e5-492c-8987-bf25dd9bc260 None admin] PATH_INFO: `/v3/services` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:45.658168 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e627dd2-52e5-492c-8987-bf25dd9bc260 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:45.658632 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e627dd2-52e5-492c-8987-bf25dd9bc260 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:45.660163 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e627dd2-52e5-492c-8987-bf25dd9bc260 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:45.672787 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 459/935] 10.222.0.33 () {56 vars in 1126 bytes} [Sun Oct 9 00:43:45 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:45.674577 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e627dd2-52e5-492c-8987-bf25dd9bc260 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:45.682378 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1e884b8-2ffe-4fa2-bb68-e5b72359775d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:45.682909 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1e884b8-2ffe-4fa2-bb68-e5b72359775d None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:45.683190 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1e884b8-2ffe-4fa2-bb68-e5b72359775d None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:45.683465 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1e884b8-2ffe-4fa2-bb68-e5b72359775d None admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:45.684157 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1e884b8-2ffe-4fa2-bb68-e5b72359775d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:45.684460 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1e884b8-2ffe-4fa2-bb68-e5b72359775d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:45.685890 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1e884b8-2ffe-4fa2-bb68-e5b72359775d None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:45.703652 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 477/936] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:43:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:45.705070 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1e884b8-2ffe-4fa2-bb68-e5b72359775d None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:45.748898 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:45.813674 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e627dd2-52e5-492c-8987-bf25dd9bc260 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:45.816221 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1e884b8-2ffe-4fa2-bb68-e5b72359775d None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:45.853014 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:45.857566 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18a25575-9211-4baa-9f84-edf9d01619ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:45.858475 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abc45a30-6fc3-463b-83df-8a59777a3045 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:45.858946 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a25575-9211-4baa-9f84-edf9d01619ec None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:45.859256 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc45a30-6fc3-463b-83df-8a59777a3045 None admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:45.859624 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc45a30-6fc3-463b-83df-8a59777a3045 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:45.860000 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc45a30-6fc3-463b-83df-8a59777a3045 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:45.860523 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a25575-9211-4baa-9f84-edf9d01619ec None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:45.860976 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a25575-9211-4baa-9f84-edf9d01619ec None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:45.861199 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abc45a30-6fc3-463b-83df-8a59777a3045 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:45.863068 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abc45a30-6fc3-463b-83df-8a59777a3045 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:45.870467 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18a25575-9211-4baa-9f84-edf9d01619ec None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:45.872229 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18a25575-9211-4baa-9f84-edf9d01619ec None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:45.887221 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-18a25575-9211-4baa-9f84-edf9d01619ec None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 09 00:43:45.887221 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:45.887221 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:45.887221 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:45.887221 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:45.887221 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:45.887221 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:45.887221 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:45.887221 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:45.887221 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:45.887221 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:45.887221 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:45.887221 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 09 00:43:45.887221 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 09 00:43:45.887221 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 09 00:43:45.887221 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 09 00:43:45.888532 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:45.888532 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:45.888532 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 09 00:43:45.888532 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 09 00:43:45.888532 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 09 00:43:45.888532 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 09 00:43:45.888532 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:45.888532 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:45.888532 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:45.888532 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:45.888532 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:45.888532 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:45.888532 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:45.888532 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:45.888532 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:45.888532 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:45.888532 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:45.888532 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:45.889513 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:45.889513 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:45.889513 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 09 00:43:45.889513 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 09 00:43:45.889513 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 09 00:43:45.889513 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 09 00:43:45.889513 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 09 00:43:45.889513 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 09 00:43:45.889513 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 09 00:43:45.889513 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:45.895993 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 460/938] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:43:45 2022] POST /identity/v3/endpoints => generated 347 bytes in 222 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:43:45.896325 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 478/938] 10.222.0.33 () {56 vars in 1117 bytes} [Sun Oct 9 00:43:45 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:45.899353 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abc45a30-6fc3-463b-83df-8a59777a3045 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:46.025535 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abc45a30-6fc3-463b-83df-8a59777a3045 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:46.082120 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bea7dcb-f37e-469d-b459-961cc5a886be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:46.082943 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bea7dcb-f37e-469d-b459-961cc5a886be None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:46.083230 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bea7dcb-f37e-469d-b459-961cc5a886be None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:46.083507 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bea7dcb-f37e-469d-b459-961cc5a886be None admin] PATH_INFO: `/v3/users` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:46.084328 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bea7dcb-f37e-469d-b459-961cc5a886be None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:46.084866 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bea7dcb-f37e-469d-b459-961cc5a886be None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:46.086121 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bea7dcb-f37e-469d-b459-961cc5a886be None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:46.107635 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 461/939] 10.222.0.33 () {56 vars in 1149 bytes} [Sun Oct 9 00:43:45 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:43:46.111866 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18a25575-9211-4baa-9f84-edf9d01619ec None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:46.223029 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:46.227102 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18a25575-9211-4baa-9f84-edf9d01619ec None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:46.263674 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2701bb0e-d65c-49d6-bdf4-91178f968957 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:46.264307 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2701bb0e-d65c-49d6-bdf4-91178f968957 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:46.264584 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2701bb0e-d65c-49d6-bdf4-91178f968957 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:46.264855 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2701bb0e-d65c-49d6-bdf4-91178f968957 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:46.276726 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2701bb0e-d65c-49d6-bdf4-91178f968957 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:46.278105 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2701bb0e-d65c-49d6-bdf4-91178f968957 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:46.292876 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-2701bb0e-d65c-49d6-bdf4-91178f968957 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:46.292876 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:46.292876 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:46.292876 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:46.292876 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:46.292876 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:46.292876 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:46.292876 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:46.292876 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:46.292876 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:46.292876 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:46.292876 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:46.292876 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:43:46.292876 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:43:46.292876 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:43:46.292876 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:43:46.293830 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:46.293830 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:46.293830 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:46.293830 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:46.293830 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:46.293830 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:46.293830 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:46.293830 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:46.293830 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:46.293830 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:46.293830 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:46.293830 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:46.293830 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:46.293830 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:46.293830 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:46.293830 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:46.293830 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:43:46.295104 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:43:46.295104 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:46.295104 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:46.295104 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 479/940] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:43:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:46.300028 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bea7dcb-f37e-469d-b459-961cc5a886be None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:46.348274 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:46.409996 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bea7dcb-f37e-469d-b459-961cc5a886be None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:46.446904 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c8877a0-7684-4eb3-aab3-f5cf0844e32a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:46.447516 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c8877a0-7684-4eb3-aab3-f5cf0844e32a None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:46.447822 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c8877a0-7684-4eb3-aab3-f5cf0844e32a None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:46.448094 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c8877a0-7684-4eb3-aab3-f5cf0844e32a None admin] PATH_INFO: `/v3/domains` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:46.448733 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c8877a0-7684-4eb3-aab3-f5cf0844e32a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:46.449029 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c8877a0-7684-4eb3-aab3-f5cf0844e32a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:46.450386 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c8877a0-7684-4eb3-aab3-f5cf0844e32a None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:46.462207 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a611c7b1-0811-45fe-8594-78ec1642e871 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:46.462518 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a611c7b1-0811-45fe-8594-78ec1642e871 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:46.462762 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a611c7b1-0811-45fe-8594-78ec1642e871 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:46.464505 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 480/941] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:43:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:43:46.465894 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 462/942] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:43:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:46.471221 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1151c1d-8162-49e3-8daa-7af8fdd0b146 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:46.471625 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1151c1d-8162-49e3-8daa-7af8fdd0b146 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:46.472017 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1151c1d-8162-49e3-8daa-7af8fdd0b146 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:46.476295 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c8877a0-7684-4eb3-aab3-f5cf0844e32a None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:46.494564 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-f1151c1d-8162-49e3-8daa-7af8fdd0b146 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:43:46.496087 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-f1151c1d-8162-49e3-8daa-7af8fdd0b146 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:43:46.579714 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 481/943] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:43:46 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 09 00:43:46.590376 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c8877a0-7684-4eb3-aab3-f5cf0844e32a None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:46.602426 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:46.628554 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a371aa2c-4694-4bdd-b6c3-5d5c4044d23d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:46.629483 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a371aa2c-4694-4bdd-b6c3-5d5c4044d23d None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:46.630032 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a371aa2c-4694-4bdd-b6c3-5d5c4044d23d None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:46.630507 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a371aa2c-4694-4bdd-b6c3-5d5c4044d23d None admin] PATH_INFO: `/v3/projects/service` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:46.638285 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a371aa2c-4694-4bdd-b6c3-5d5c4044d23d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:46.640049 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a371aa2c-4694-4bdd-b6c3-5d5c4044d23d None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:46.651641 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-a371aa2c-4694-4bdd-b6c3-5d5c4044d23d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 09 00:43:46.651641 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:46.651641 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:46.651641 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:46.651641 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:46.651641 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:46.651641 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:46.651641 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:46.651641 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:46.651641 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:46.651641 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:46.651641 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:46.651641 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 09 00:43:46.651641 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 09 00:43:46.651641 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 09 00:43:46.652526 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 09 00:43:46.652526 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:46.652526 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:46.652526 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:46.652526 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:46.652526 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:46.652526 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:46.652526 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:46.652526 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:46.652526 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:46.652526 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:46.652526 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:46.652526 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:46.652526 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:46.652526 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:46.652526 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:46.652526 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:46.653420 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 09 00:43:46.653420 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 09 00:43:46.653420 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 09 00:43:46.653420 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 09 00:43:46.653420 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 09 00:43:46.653420 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 09 00:43:46.653420 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 09 00:43:46.653420 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:46.655786 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 463/944] 10.222.0.33 () {56 vars in 1111 bytes} [Sun Oct 9 00:43:46 2022] GET /identity/v3/projects/service => generated 88 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 09 00:43:46.661804 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1151c1d-8162-49e3-8daa-7af8fdd0b146 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:46.748117 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:46.770232 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1151c1d-8162-49e3-8daa-7af8fdd0b146 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:46.806825 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e77af30-af0e-4774-b010-307994a61bb2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:46.807834 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e77af30-af0e-4774-b010-307994a61bb2 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:46.808353 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e77af30-af0e-4774-b010-307994a61bb2 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:46.808841 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e77af30-af0e-4774-b010-307994a61bb2 None admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:46.809741 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e77af30-af0e-4774-b010-307994a61bb2 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:46.810254 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e77af30-af0e-4774-b010-307994a61bb2 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:46.811928 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e77af30-af0e-4774-b010-307994a61bb2 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:46.825464 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 482/945] 10.222.0.33 () {56 vars in 1148 bytes} [Sun Oct 9 00:43:46 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:46.831862 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a371aa2c-4694-4bdd-b6c3-5d5c4044d23d None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:46.842582 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e77af30-af0e-4774-b010-307994a61bb2 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:46.953737 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a371aa2c-4694-4bdd-b6c3-5d5c4044d23d None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:46.954241 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e77af30-af0e-4774-b010-307994a61bb2 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:46.992211 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a15a4e09-ce6a-4638-a83d-44a139cb7fc0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:46.993399 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a15a4e09-ce6a-4638-a83d-44a139cb7fc0 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:46.993933 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a15a4e09-ce6a-4638-a83d-44a139cb7fc0 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:46.994145 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76befb8a-7e6c-4140-96e6-f16687cbdae8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:46.994695 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a15a4e09-ce6a-4638-a83d-44a139cb7fc0 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:46.994816 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76befb8a-7e6c-4140-96e6-f16687cbdae8 None admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:46.995104 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76befb8a-7e6c-4140-96e6-f16687cbdae8 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:46.995380 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76befb8a-7e6c-4140-96e6-f16687cbdae8 None admin] PATH_INFO: `/v3/projects/013c83476b1340fbb051061833c4a9c6/users/5f3247ddcda54394995d844c776bf43c/roles/fddb7718710e48bb8034bd9390b820a2` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:46.996118 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a15a4e09-ce6a-4638-a83d-44a139cb7fc0 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:46.997669 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76befb8a-7e6c-4140-96e6-f16687cbdae8 None admin] RBAC: Authorizing `identity:create_grant(project_id=013c83476b1340fbb051061833c4a9c6, user_id=5f3247ddcda54394995d844c776bf43c, role_id=fddb7718710e48bb8034bd9390b820a2)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:46.998052 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a15a4e09-ce6a-4638-a83d-44a139cb7fc0 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:46.999114 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76befb8a-7e6c-4140-96e6-f16687cbdae8 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:47.008765 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-a15a4e09-ce6a-4638-a83d-44a139cb7fc0 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 09 00:43:47.008765 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:47.008765 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 09 00:43:47.008765 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 09 00:43:47.008765 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 09 00:43:47.008765 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 09 00:43:47.008765 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 09 00:43:47.008765 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 09 00:43:47.008765 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 09 00:43:47.008765 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:47.008765 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 09 00:43:47.008765 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:47.008765 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:47.008765 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:47.008765 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:47.008765 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:47.008765 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:47.009821 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:47.009821 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:47.009821 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:47.009821 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:47.009821 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:47.009821 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:47.009821 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 09 00:43:47.009821 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 09 00:43:47.009821 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 09 00:43:47.009821 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 09 00:43:47.009821 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:47.009821 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:47.009821 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:47.009821 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:47.009821 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:47.009821 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:47.010714 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:47.010714 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:47.010714 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:47.010714 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:47.010714 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:47.010714 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:47.010714 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:47.010714 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:47.010714 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:47.010714 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:47.010714 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 09 00:43:47.010714 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 09 00:43:47.010714 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 09 00:43:47.010714 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:47.013652 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 483/946] 10.222.0.33 () {56 vars in 1126 bytes} [Sun Oct 9 00:43:46 2022] GET /identity/v3/services/object-store => generated 93 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:47.020944 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a15a4e09-ce6a-4638-a83d-44a139cb7fc0 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:47.023954 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 464/947] 10.222.0.33 () {58 vars in 1439 bytes} [Sun Oct 9 00:43:46 2022] PUT /identity/v3/projects/013c83476b1340fbb051061833c4a9c6/users/5f3247ddcda54394995d844c776bf43c/roles/fddb7718710e48bb8034bd9390b820a2 => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:43:47.033365 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:47.157651 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a15a4e09-ce6a-4638-a83d-44a139cb7fc0 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:47.198488 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abe5a3d0-0c5d-4c27-8c3d-de087f009fee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:47.199460 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe5a3d0-0c5d-4c27-8c3d-de087f009fee None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:47.200009 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe5a3d0-0c5d-4c27-8c3d-de087f009fee None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:47.200479 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe5a3d0-0c5d-4c27-8c3d-de087f009fee None admin] PATH_INFO: `/v3/services` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:47.201333 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abe5a3d0-0c5d-4c27-8c3d-de087f009fee None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:47.201901 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abe5a3d0-0c5d-4c27-8c3d-de087f009fee None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:47.203685 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abe5a3d0-0c5d-4c27-8c3d-de087f009fee None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:47.209716 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 484/948] 10.222.0.33 () {56 vars in 1122 bytes} [Sun Oct 9 00:43:47 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:43:47.216702 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76befb8a-7e6c-4140-96e6-f16687cbdae8 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:47.248936 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:47.328423 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76befb8a-7e6c-4140-96e6-f16687cbdae8 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:47.348916 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd3740f-e2ac-48cb-9290-e07e8e82270f None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:47.349237 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd3740f-e2ac-48cb-9290-e07e8e82270f None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:47.349541 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd3740f-e2ac-48cb-9290-e07e8e82270f None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:47.350930 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 485/949] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:43:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:43:47.356046 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f2021b8-ffd9-4e28-9abf-34625f0810c6 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:47.356331 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f2021b8-ffd9-4e28-9abf-34625f0810c6 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:47.356605 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f2021b8-ffd9-4e28-9abf-34625f0810c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:47.364224 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab7dad0b-d16c-42bb-b633-304510aee591 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:47.364958 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab7dad0b-d16c-42bb-b633-304510aee591 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:47.365347 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab7dad0b-d16c-42bb-b633-304510aee591 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:47.365716 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab7dad0b-d16c-42bb-b633-304510aee591 None admin] PATH_INFO: `/v3/services` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:47.366982 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab7dad0b-d16c-42bb-b633-304510aee591 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:47.367386 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab7dad0b-d16c-42bb-b633-304510aee591 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:47.368915 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab7dad0b-d16c-42bb-b633-304510aee591 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:47.374231 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 465/950] 10.222.0.33 () {56 vars in 1122 bytes} [Sun Oct 9 00:43:47 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:47.375706 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-6f2021b8-ffd9-4e28-9abf-34625f0810c6 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:43:47.376892 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-6f2021b8-ffd9-4e28-9abf-34625f0810c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:43:47.380522 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab7dad0b-d16c-42bb-b633-304510aee591 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:47.444700 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:47.455726 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 486/951] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:43:47 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 09 00:43:47.473610 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:47.489198 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab7dad0b-d16c-42bb-b633-304510aee591 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:47.524414 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-565e2cb6-5505-4046-99ff-9a984096d215 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:47.525281 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-565e2cb6-5505-4046-99ff-9a984096d215 None admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:47.525748 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-565e2cb6-5505-4046-99ff-9a984096d215 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:47.526157 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-565e2cb6-5505-4046-99ff-9a984096d215 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:47.526946 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-565e2cb6-5505-4046-99ff-9a984096d215 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:47.528532 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-565e2cb6-5505-4046-99ff-9a984096d215 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:47.552912 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 466/952] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:43:47 2022] POST /identity/v3/endpoints => generated 349 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:43:47.740291 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f2021b8-ffd9-4e28-9abf-34625f0810c6 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:47.844960 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f2021b8-ffd9-4e28-9abf-34625f0810c6 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:47.886421 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e735179-c040-4385-a878-f2bc3e99687e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:47.887036 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e735179-c040-4385-a878-f2bc3e99687e None admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:47.887315 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e735179-c040-4385-a878-f2bc3e99687e None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:47.887602 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e735179-c040-4385-a878-f2bc3e99687e None admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:47.888245 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e735179-c040-4385-a878-f2bc3e99687e None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:47.889670 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e735179-c040-4385-a878-f2bc3e99687e None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:47.892274 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: WARNING keystone.server.flask.common [None req-4e735179-c040-4385-a878-f2bc3e99687e None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Oct 09 00:43:47.912455 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.sql.core [None req-4e735179-c040-4385-a878-f2bc3e99687e None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Oct 09 00:43:47.912455 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 09 00:43:47.912455 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [parameters: {'id': '2371519dcf3f45c9ae1525b658bc2311', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Oct 09 00:43:47.912455 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81452) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 09 00:43:47.931110 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-4e735179-c040-4385-a878-f2bc3e99687e None admin] Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 2371519dcf3f45c9ae1525b658bc2311.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 2371519dcf3f45c9ae1525b658bc2311. Oct 09 00:43:47.931110 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:47.931110 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 09 00:43:47.931110 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 09 00:43:47.931110 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 09 00:43:47.931110 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 09 00:43:47.931110 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 09 00:43:47.931110 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application result = self._query(query) Oct 09 00:43:47.931110 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 09 00:43:47.931110 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application conn.query(q) Oct 09 00:43:47.931110 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 09 00:43:47.931110 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 09 00:43:47.931110 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 09 00:43:47.932097 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application result.read() Oct 09 00:43:47.932097 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 09 00:43:47.932097 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 09 00:43:47.932097 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 09 00:43:47.932097 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 09 00:43:47.932097 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 09 00:43:47.932097 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 09 00:43:47.932097 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 09 00:43:47.932097 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 09 00:43:47.932097 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Oct 09 00:43:47.932097 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:47.932097 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 09 00:43:47.932097 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:47.932097 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:47.932097 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 09 00:43:47.932097 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 09 00:43:47.932097 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Oct 09 00:43:47.932097 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return project_ref.to_dict() Oct 09 00:43:47.933004 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 09 00:43:47.933004 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application next(self.gen) Oct 09 00:43:47.933004 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Oct 09 00:43:47.933004 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application yield resource Oct 09 00:43:47.933004 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 09 00:43:47.933004 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application next(self.gen) Oct 09 00:43:47.933004 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Oct 09 00:43:47.933004 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self.session.rollback() Oct 09 00:43:47.933004 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 09 00:43:47.933004 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self.force_reraise() Oct 09 00:43:47.933004 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 09 00:43:47.933004 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise self.value Oct 09 00:43:47.933004 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Oct 09 00:43:47.933004 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 09 00:43:47.933004 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Oct 09 00:43:47.933004 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application session.commit() Oct 09 00:43:47.933004 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 09 00:43:47.933849 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 09 00:43:47.933849 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 09 00:43:47.933849 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self._prepare_impl() Oct 09 00:43:47.933849 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 09 00:43:47.933849 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self.session.flush() Oct 09 00:43:47.933849 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 09 00:43:47.933849 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self._flush(objects) Oct 09 00:43:47.933849 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 09 00:43:47.933849 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 09 00:43:47.933849 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 09 00:43:47.933849 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application compat.raise_( Oct 09 00:43:47.933849 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 09 00:43:47.933849 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception Oct 09 00:43:47.933849 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 09 00:43:47.933849 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application flush_context.execute() Oct 09 00:43:47.933849 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 09 00:43:47.933849 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rec.execute(self) Oct 09 00:43:47.934714 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 09 00:43:47.934714 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 09 00:43:47.934714 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 09 00:43:47.934714 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 09 00:43:47.934714 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Oct 09 00:43:47.934714 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application c = connection._execute_20( Oct 09 00:43:47.934714 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 09 00:43:47.934714 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 09 00:43:47.934714 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 09 00:43:47.934714 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 09 00:43:47.934714 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 09 00:43:47.934714 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 09 00:43:47.934714 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 09 00:43:47.934714 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 09 00:43:47.934714 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 09 00:43:47.935496 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 09 00:43:47.935496 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 09 00:43:47.935496 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception Oct 09 00:43:47.935496 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 09 00:43:47.935496 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 09 00:43:47.935496 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 09 00:43:47.935496 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 09 00:43:47.935496 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 09 00:43:47.935496 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application result = self._query(query) Oct 09 00:43:47.935496 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 09 00:43:47.935496 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application conn.query(q) Oct 09 00:43:47.935496 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 09 00:43:47.935496 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 09 00:43:47.935496 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 09 00:43:47.935496 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application result.read() Oct 09 00:43:47.935496 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 09 00:43:47.935496 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 09 00:43:47.936838 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 09 00:43:47.936838 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 09 00:43:47.936838 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 09 00:43:47.936838 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 09 00:43:47.936838 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 09 00:43:47.936838 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 09 00:43:47.936838 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Oct 09 00:43:47.936838 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 09 00:43:47.936838 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [parameters: {'id': '2371519dcf3f45c9ae1525b658bc2311', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Oct 09 00:43:47.936838 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 09 00:43:47.936838 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:47.936838 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 09 00:43:47.936838 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:47.936838 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:47.936838 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Oct 09 00:43:47.938350 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Oct 09 00:43:47.938350 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 09 00:43:47.938350 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 09 00:43:47.938350 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name service at domain ID default. Oct 09 00:43:47.938350 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:47.938350 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 09 00:43:47.938350 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:47.938350 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:47.938350 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:47.938350 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:47.938350 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:47.938350 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:47.938350 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:47.938350 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:47.938350 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:47.938350 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:47.938350 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:47.939436 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:47.939436 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 176, in post Oct 09 00:43:47.939436 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_project( Oct 09 00:43:47.939436 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:47.939436 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:47.939436 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Oct 09 00:43:47.939436 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.Conflict( Oct 09 00:43:47.939436 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 2371519dcf3f45c9ae1525b658bc2311. Oct 09 00:43:47.939436 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:47.939436 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 487/953] 10.222.0.33 () {60 vars in 1140 bytes} [Sun Oct 9 00:43:47 2022] POST /identity/v3/projects => generated 262 bytes in 193 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:47.940826 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-565e2cb6-5505-4046-99ff-9a984096d215 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:48.049766 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-565e2cb6-5505-4046-99ff-9a984096d215 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:48.090551 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62ca3287-108d-4ed2-a8e6-c6a35622e4ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:48.091440 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ca3287-108d-4ed2-a8e6-c6a35622e4ac None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:48.091944 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ca3287-108d-4ed2-a8e6-c6a35622e4ac None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:48.092406 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ca3287-108d-4ed2-a8e6-c6a35622e4ac None admin] PATH_INFO: `/v3/projects/service` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:48.099378 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62ca3287-108d-4ed2-a8e6-c6a35622e4ac None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:48.101568 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62ca3287-108d-4ed2-a8e6-c6a35622e4ac None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:48.112736 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-62ca3287-108d-4ed2-a8e6-c6a35622e4ac None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 09 00:43:48.112736 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:48.112736 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:48.112736 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:48.112736 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:48.112736 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:48.112736 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:48.112736 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:48.112736 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:48.112736 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:48.112736 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:48.112736 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:48.112736 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 09 00:43:48.112736 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 09 00:43:48.112736 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 09 00:43:48.113695 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 09 00:43:48.113695 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:48.113695 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:48.113695 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:48.113695 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:48.113695 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:48.113695 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:48.113695 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:48.113695 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:48.113695 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:48.113695 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:48.113695 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:48.113695 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:48.113695 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:48.113695 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:48.113695 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:48.113695 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:48.114639 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 09 00:43:48.114639 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 09 00:43:48.114639 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 09 00:43:48.114639 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 09 00:43:48.114639 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 09 00:43:48.114639 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 09 00:43:48.114639 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 09 00:43:48.114639 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:48.114639 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 467/954] 10.222.0.33 () {56 vars in 1111 bytes} [Sun Oct 9 00:43:47 2022] GET /identity/v3/projects/service => generated 88 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:48.121927 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e735179-c040-4385-a878-f2bc3e99687e None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:48.175809 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:48.228971 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e735179-c040-4385-a878-f2bc3e99687e None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:48.269797 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-917ba21a-85f5-480b-bc85-84bea254ed49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:48.270458 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-917ba21a-85f5-480b-bc85-84bea254ed49 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:48.270742 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-917ba21a-85f5-480b-bc85-84bea254ed49 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:48.271018 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-917ba21a-85f5-480b-bc85-84bea254ed49 None admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:48.271712 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-917ba21a-85f5-480b-bc85-84bea254ed49 None admin] RBAC: Adding query filter params (name=service) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:48.272019 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-917ba21a-85f5-480b-bc85-84bea254ed49 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:48.274236 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-917ba21a-85f5-480b-bc85-84bea254ed49 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:48.296124 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 488/955] 10.222.0.33 () {56 vars in 1112 bytes} [Sun Oct 9 00:43:48 2022] GET /identity/v3/projects?name=service => generated 411 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:48.363151 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8e7cb3-c59a-41c9-8ecd-2e9f72d21cdc None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:48.363676 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8e7cb3-c59a-41c9-8ecd-2e9f72d21cdc None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:48.363676 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8e7cb3-c59a-41c9-8ecd-2e9f72d21cdc None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:48.365262 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 468/956] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:43:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:43:48.371187 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a37d363-9818-4900-a816-9f34d5ade7b1 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:48.371505 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a37d363-9818-4900-a816-9f34d5ade7b1 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:48.371793 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a37d363-9818-4900-a816-9f34d5ade7b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:48.396098 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-9a37d363-9818-4900-a816-9f34d5ade7b1 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:43:48.398196 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-9a37d363-9818-4900-a816-9f34d5ade7b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:43:48.474981 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 489/957] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:43:48 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 09 00:43:48.754595 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e8e7cb3-c59a-41c9-8ecd-2e9f72d21cdc None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:48.758232 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:48.867015 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e8e7cb3-c59a-41c9-8ecd-2e9f72d21cdc None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:48.902451 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eebcecd-94d9-4d58-8bd4-51650012a498 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:48.903085 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eebcecd-94d9-4d58-8bd4-51650012a498 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:48.903360 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eebcecd-94d9-4d58-8bd4-51650012a498 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:48.903718 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eebcecd-94d9-4d58-8bd4-51650012a498 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:48.904299 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eebcecd-94d9-4d58-8bd4-51650012a498 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:48.905653 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eebcecd-94d9-4d58-8bd4-51650012a498 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:48.914914 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-3eebcecd-94d9-4d58-8bd4-51650012a498 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Oct 09 00:43:48.914914 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:48.914914 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 09 00:43:48.914914 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 09 00:43:48.914914 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 09 00:43:48.914914 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 09 00:43:48.914914 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 09 00:43:48.914914 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 09 00:43:48.914914 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 09 00:43:48.914914 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:48.914914 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 09 00:43:48.914914 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:48.914914 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:48.914914 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:48.914914 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:48.914914 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:48.914914 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:48.916021 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:48.916021 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:48.916021 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:48.916021 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:48.916021 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:48.916021 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:48.916021 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 09 00:43:48.916021 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 09 00:43:48.916021 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 09 00:43:48.916021 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 09 00:43:48.916021 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:48.916021 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:48.916021 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:48.916021 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:48.916021 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:48.916021 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:48.916875 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:48.916875 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:48.916875 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:48.916875 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:48.916875 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:48.916875 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:48.916875 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:48.916875 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:48.916875 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:48.916875 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:48.916875 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 09 00:43:48.916875 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 09 00:43:48.916875 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 09 00:43:48.916875 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:48.916875 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 469/958] 10.222.0.33 () {56 vars in 1111 bytes} [Sun Oct 9 00:43:48 2022] GET /identity/v3/services/compute => generated 88 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:48.923912 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a37d363-9818-4900-a816-9f34d5ade7b1 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:48.969812 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:49.029489 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a37d363-9818-4900-a816-9f34d5ade7b1 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:49.064629 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b5ee447-b929-4c51-a580-bc51815eec56 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:49.065231 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5ee447-b929-4c51-a580-bc51815eec56 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:49.065510 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5ee447-b929-4c51-a580-bc51815eec56 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:49.065778 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5ee447-b929-4c51-a580-bc51815eec56 None admin] PATH_INFO: `/v3/services` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:49.066397 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b5ee447-b929-4c51-a580-bc51815eec56 None admin] RBAC: Adding query filter params (name=compute) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:49.066694 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b5ee447-b929-4c51-a580-bc51815eec56 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:49.068101 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b5ee447-b929-4c51-a580-bc51815eec56 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:49.084011 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 490/959] 10.222.0.33 () {56 vars in 1112 bytes} [Sun Oct 9 00:43:48 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:49.089855 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eebcecd-94d9-4d58-8bd4-51650012a498 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:49.160433 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:49.198549 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eebcecd-94d9-4d58-8bd4-51650012a498 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:49.238586 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a036dcb0-d7ff-47a0-911e-8e6ec12b6485 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:49.239460 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a036dcb0-d7ff-47a0-911e-8e6ec12b6485 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:49.239948 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a036dcb0-d7ff-47a0-911e-8e6ec12b6485 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:49.240369 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a036dcb0-d7ff-47a0-911e-8e6ec12b6485 None admin] PATH_INFO: `/v3/services` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:49.241713 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a036dcb0-d7ff-47a0-911e-8e6ec12b6485 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:49.242180 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a036dcb0-d7ff-47a0-911e-8e6ec12b6485 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:49.243782 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a036dcb0-d7ff-47a0-911e-8e6ec12b6485 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:49.249414 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 470/960] 10.222.0.33 () {56 vars in 1112 bytes} [Sun Oct 9 00:43:49 2022] GET /identity/v3/services?type=compute => generated 124 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:49.337993 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfdfe852-515e-4d01-b982-5b582dae1ba5 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:49.338507 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfdfe852-515e-4d01-b982-5b582dae1ba5 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:49.338680 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfdfe852-515e-4d01-b982-5b582dae1ba5 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:49.340508 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 491/961] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:43:49 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:43:49.346466 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb126dbc-265e-4898-81a7-cfca93f2c7f5 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:49.346896 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb126dbc-265e-4898-81a7-cfca93f2c7f5 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:49.347277 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb126dbc-265e-4898-81a7-cfca93f2c7f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:49.368165 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-bb126dbc-265e-4898-81a7-cfca93f2c7f5 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:43:49.369381 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-bb126dbc-265e-4898-81a7-cfca93f2c7f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:43:49.448983 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 471/962] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:43:49 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 09 00:43:49.721450 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfdfe852-515e-4d01-b982-5b582dae1ba5 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:49.815992 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857b8348-0a2f-4203-8705-ab657b1b915d None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:49.816320 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857b8348-0a2f-4203-8705-ab657b1b915d None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:49.816582 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857b8348-0a2f-4203-8705-ab657b1b915d None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:49.818184 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 472/963] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:43:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 09 00:43:49.824446 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb95ce2-968f-481a-afcc-f33591cf6e1f None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:49.824730 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb95ce2-968f-481a-afcc-f33591cf6e1f None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:49.825007 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb95ce2-968f-481a-afcc-f33591cf6e1f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:49.829232 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfdfe852-515e-4d01-b982-5b582dae1ba5 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:49.832140 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:49.844078 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-6cb95ce2-968f-481a-afcc-f33591cf6e1f None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:43:49.845650 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-6cb95ce2-968f-481a-afcc-f33591cf6e1f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:43:49.873097 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e671087-d6b5-433c-a45f-74e9bea63ce9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:49.873731 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e671087-d6b5-433c-a45f-74e9bea63ce9 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:49.874012 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e671087-d6b5-433c-a45f-74e9bea63ce9 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:49.874286 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e671087-d6b5-433c-a45f-74e9bea63ce9 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:49.888635 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e671087-d6b5-433c-a45f-74e9bea63ce9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:43:49.894471 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-6e671087-d6b5-433c-a45f-74e9bea63ce9 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 09 00:43:49.894471 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:49.894471 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:49.894471 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:49.894471 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:49.894471 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:49.894471 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:49.894471 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:49.894471 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:49.894471 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:49.894471 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:49.894471 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:49.894471 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:43:49.894471 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:43:49.894471 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:43:49.894471 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:43:49.895418 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:43:49.895418 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:43:49.895418 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:43:49.895418 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:43:49.895418 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:49.895418 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:49.895418 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:49.895418 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:49.895418 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:49.895418 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:49.895418 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:49.895418 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:49.895418 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:49.895418 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:49.895418 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:49.895418 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:49.896299 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:49.896299 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:49.896299 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:49.896299 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:49.896299 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:43:49.896299 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:43:49.896299 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:43:49.896299 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:43:49.896299 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:43:49.896299 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:43:49.896299 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 09 00:43:49.896299 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:49.900795 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 492/964] 10.222.0.33 () {56 vars in 1120 bytes} [Sun Oct 9 00:43:49 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:49.901821 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e671087-d6b5-433c-a45f-74e9bea63ce9 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:49.914620 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:49.933723 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 473/965] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:43:49 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 09 00:43:50.011027 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e671087-d6b5-433c-a45f-74e9bea63ce9 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:50.046515 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c34e7730-6131-451c-98d8-03fde1e70a6d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:50.047131 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34e7730-6131-451c-98d8-03fde1e70a6d None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:50.047414 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34e7730-6131-451c-98d8-03fde1e70a6d None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:50.047716 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34e7730-6131-451c-98d8-03fde1e70a6d None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:50.048384 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c34e7730-6131-451c-98d8-03fde1e70a6d None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:50.048687 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c34e7730-6131-451c-98d8-03fde1e70a6d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:50.050062 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c34e7730-6131-451c-98d8-03fde1e70a6d None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:50.063880 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 493/966] 10.222.0.33 () {56 vars in 1115 bytes} [Sun Oct 9 00:43:49 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:43:50.066288 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cb95ce2-968f-481a-afcc-f33591cf6e1f None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:50.174252 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cb95ce2-968f-481a-afcc-f33591cf6e1f None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:50.210681 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81afbd59-41a5-4ecd-82b7-702086e1bd18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:50.211328 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81afbd59-41a5-4ecd-82b7-702086e1bd18 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:50.211621 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81afbd59-41a5-4ecd-82b7-702086e1bd18 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:50.211971 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81afbd59-41a5-4ecd-82b7-702086e1bd18 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:50.216712 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c34e7730-6131-451c-98d8-03fde1e70a6d None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:50.225214 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81afbd59-41a5-4ecd-82b7-702086e1bd18 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:50.227336 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81afbd59-41a5-4ecd-82b7-702086e1bd18 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:50.245119 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-81afbd59-41a5-4ecd-82b7-702086e1bd18 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:50.245119 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:50.245119 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:50.245119 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:50.245119 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:50.245119 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:50.245119 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:50.245119 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:50.245119 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:50.245119 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:50.245119 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:50.245119 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:50.245119 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:43:50.245119 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:43:50.245119 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:43:50.245119 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:43:50.246137 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:50.246137 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:50.246137 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:50.246137 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:50.246137 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:50.246137 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:50.246137 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:50.246137 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:50.246137 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:50.246137 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:50.246137 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:50.246137 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:50.246137 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:50.246137 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:50.246137 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:50.246137 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:50.246137 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:43:50.247073 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:43:50.247073 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:50.247073 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:50.251180 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 474/967] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:43:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:50.252909 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81afbd59-41a5-4ecd-82b7-702086e1bd18 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:50.254930 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:50.306769 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:50.358735 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c34e7730-6131-451c-98d8-03fde1e70a6d None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:50.378100 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81afbd59-41a5-4ecd-82b7-702086e1bd18 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:50.405863 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b84afd5d-bff6-49b8-8b00-3ae745537571 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:50.407090 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b84afd5d-bff6-49b8-8b00-3ae745537571 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:50.407860 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b84afd5d-bff6-49b8-8b00-3ae745537571 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:50.408783 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b84afd5d-bff6-49b8-8b00-3ae745537571 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:50.409916 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b84afd5d-bff6-49b8-8b00-3ae745537571 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:50.412477 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b84afd5d-bff6-49b8-8b00-3ae745537571 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:50.417993 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f70f3729-57cd-401d-bd08-df5aa32ae664 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:50.418604 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70f3729-57cd-401d-bd08-df5aa32ae664 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:50.418886 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70f3729-57cd-401d-bd08-df5aa32ae664 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:50.419162 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70f3729-57cd-401d-bd08-df5aa32ae664 None admin] PATH_INFO: `/v3/domains` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:50.419841 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f70f3729-57cd-401d-bd08-df5aa32ae664 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:50.420145 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f70f3729-57cd-401d-bd08-df5aa32ae664 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:50.421492 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f70f3729-57cd-401d-bd08-df5aa32ae664 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:50.425464 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-b84afd5d-bff6-49b8-8b00-3ae745537571 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 09 00:43:50.425464 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:50.425464 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 09 00:43:50.425464 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 09 00:43:50.425464 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 09 00:43:50.425464 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 09 00:43:50.425464 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 09 00:43:50.425464 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 09 00:43:50.425464 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 09 00:43:50.425464 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:50.425464 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 09 00:43:50.425464 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:50.425464 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:50.425464 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:50.425464 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:50.425464 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:50.425464 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:50.426457 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:50.426457 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:50.426457 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:50.426457 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:50.426457 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:50.426457 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:50.426457 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 09 00:43:50.426457 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 09 00:43:50.426457 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 09 00:43:50.426457 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 09 00:43:50.426457 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:50.426457 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:50.426457 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:50.426457 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:50.426457 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:50.426457 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:50.427288 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:50.427288 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:50.427288 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:50.427288 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:50.427288 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:50.427288 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:50.427288 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:50.427288 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:50.427288 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:50.427288 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:50.427288 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 09 00:43:50.427288 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 09 00:43:50.427288 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 09 00:43:50.427288 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:50.435544 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 494/968] 10.222.0.33 () {56 vars in 1126 bytes} [Sun Oct 9 00:43:50 2022] GET /identity/v3/services/object-store => generated 93 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 09 00:43:50.436841 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b84afd5d-bff6-49b8-8b00-3ae745537571 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:50.444898 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 475/969] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:43:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:50.446208 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f70f3729-57cd-401d-bd08-df5aa32ae664 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:50.567285 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f70f3729-57cd-401d-bd08-df5aa32ae664 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:50.568784 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:50.574694 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:50.586668 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b84afd5d-bff6-49b8-8b00-3ae745537571 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:50.612884 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea5d21b-89be-47ee-9cb1-63b860773e1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:50.613524 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea5d21b-89be-47ee-9cb1-63b860773e1e None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:50.613809 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea5d21b-89be-47ee-9cb1-63b860773e1e None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:50.614086 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea5d21b-89be-47ee-9cb1-63b860773e1e None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:50.621776 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eea5d21b-89be-47ee-9cb1-63b860773e1e None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:50.622691 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5198ff9-38c8-47b0-9369-62ac8620f87e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:50.623211 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eea5d21b-89be-47ee-9cb1-63b860773e1e None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:50.623331 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5198ff9-38c8-47b0-9369-62ac8620f87e None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:50.623615 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5198ff9-38c8-47b0-9369-62ac8620f87e None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:50.623871 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5198ff9-38c8-47b0-9369-62ac8620f87e None admin] PATH_INFO: `/v3/services` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:50.624499 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5198ff9-38c8-47b0-9369-62ac8620f87e None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:50.624911 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5198ff9-38c8-47b0-9369-62ac8620f87e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:50.626239 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5198ff9-38c8-47b0-9369-62ac8620f87e None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:50.637393 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 495/970] 10.222.0.33 () {56 vars in 1122 bytes} [Sun Oct 9 00:43:50 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:50.637393 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c7b2e8-d45f-4155-8ea0-b8f66ded2387 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:50.637393 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c7b2e8-d45f-4155-8ea0-b8f66ded2387 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:50.637393 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c7b2e8-d45f-4155-8ea0-b8f66ded2387 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:50.637393 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 496/971] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:43:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:43:50.637829 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-eea5d21b-89be-47ee-9cb1-63b860773e1e None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 09 00:43:50.637829 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:50.637829 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:50.637829 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:50.637829 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:50.637829 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:50.637829 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:50.637829 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:50.637829 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:50.637829 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:50.637829 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:50.637829 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:50.637829 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 09 00:43:50.637829 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 09 00:43:50.637829 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 09 00:43:50.637829 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 09 00:43:50.640358 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:50.640358 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:50.640358 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 09 00:43:50.640358 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 09 00:43:50.640358 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 09 00:43:50.640358 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 09 00:43:50.640358 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:50.640358 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:50.640358 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:50.640358 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:50.640358 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:50.640358 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:50.640358 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:50.640358 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:50.640358 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:50.640358 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:50.640358 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:50.640358 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:50.641530 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:50.641530 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:50.641530 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 09 00:43:50.641530 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 09 00:43:50.641530 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 09 00:43:50.641530 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 09 00:43:50.641530 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 09 00:43:50.641530 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 09 00:43:50.641530 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 09 00:43:50.641530 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:50.641530 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 476/972] 10.222.0.33 () {56 vars in 1117 bytes} [Sun Oct 9 00:43:50 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:50.643560 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61c7b2e8-d45f-4155-8ea0-b8f66ded2387 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:50.644105 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a8e525f-244b-4067-94d7-513a734961ee None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:50.644392 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a8e525f-244b-4067-94d7-513a734961ee None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:50.644685 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a8e525f-244b-4067-94d7-513a734961ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:50.664695 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-7a8e525f-244b-4067-94d7-513a734961ee None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:43:50.665871 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-7a8e525f-244b-4067-94d7-513a734961ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:43:50.736761 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 477/973] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:43:50 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 09 00:43:50.737990 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a8e525f-244b-4067-94d7-513a734961ee None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:50.761431 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61c7b2e8-d45f-4155-8ea0-b8f66ded2387 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:50.812804 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5d8272b-0e00-47ae-bda3-9ff422051a6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:50.814332 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d8272b-0e00-47ae-bda3-9ff422051a6e None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:50.814686 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d8272b-0e00-47ae-bda3-9ff422051a6e None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:50.815038 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d8272b-0e00-47ae-bda3-9ff422051a6e None admin] PATH_INFO: `/v3/services` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:50.815778 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5d8272b-0e00-47ae-bda3-9ff422051a6e None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:50.816161 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5d8272b-0e00-47ae-bda3-9ff422051a6e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:50.817663 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5d8272b-0e00-47ae-bda3-9ff422051a6e None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:50.829003 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 497/974] 10.222.0.33 () {56 vars in 1122 bytes} [Sun Oct 9 00:43:50 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:43:50.830756 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5d8272b-0e00-47ae-bda3-9ff422051a6e None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:50.841650 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:50.853054 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a8e525f-244b-4067-94d7-513a734961ee None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:50.856338 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:50.892306 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2406f7b9-904a-473e-8411-7a91c357ec68 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:50.893287 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2406f7b9-904a-473e-8411-7a91c357ec68 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:50.893775 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2406f7b9-904a-473e-8411-7a91c357ec68 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:50.894254 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2406f7b9-904a-473e-8411-7a91c357ec68 None admin] PATH_INFO: `/v3/users` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:50.895156 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2406f7b9-904a-473e-8411-7a91c357ec68 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:50.895691 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2406f7b9-904a-473e-8411-7a91c357ec68 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:50.898048 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2406f7b9-904a-473e-8411-7a91c357ec68 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:50.922121 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 478/975] 10.222.0.33 () {56 vars in 1149 bytes} [Sun Oct 9 00:43:50 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:50.923728 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2406f7b9-904a-473e-8411-7a91c357ec68 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:50.960752 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5d8272b-0e00-47ae-bda3-9ff422051a6e None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:50.997860 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4dc50d7-befb-44ad-bed1-b3f1c78200ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:50.998838 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4dc50d7-befb-44ad-bed1-b3f1c78200ef None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:50.999423 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4dc50d7-befb-44ad-bed1-b3f1c78200ef None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:50.999942 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4dc50d7-befb-44ad-bed1-b3f1c78200ef None admin] PATH_INFO: `/v3/endpoints` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:51.000968 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4dc50d7-befb-44ad-bed1-b3f1c78200ef None admin] RBAC: Adding query filter params (interface=admin, service_id=2eb616ab91c04b3bacba0ee7173bf8c1, region_id=RegionOne) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:51.001518 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4dc50d7-befb-44ad-bed1-b3f1c78200ef None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:51.003233 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4dc50d7-befb-44ad-bed1-b3f1c78200ef None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:51.014360 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 498/976] 10.222.0.33 () {56 vars in 1249 bytes} [Sun Oct 9 00:43:50 2022] GET /identity/v3/endpoints?service_id=2eb616ab91c04b3bacba0ee7173bf8c1&interface=admin®ion_id=RegionOne => generated 193 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:51.015456 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4dc50d7-befb-44ad-bed1-b3f1c78200ef None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:51.036843 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2406f7b9-904a-473e-8411-7a91c357ec68 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:51.079907 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a94f3621-c5f1-4f26-8eef-d7ded353a325 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:51.080257 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94f3621-c5f1-4f26-8eef-d7ded353a325 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:51.080528 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94f3621-c5f1-4f26-8eef-d7ded353a325 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:51.080792 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94f3621-c5f1-4f26-8eef-d7ded353a325 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:51.103750 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:51.109449 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:51.129039 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4dc50d7-befb-44ad-bed1-b3f1c78200ef None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:51.158784 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a94f3621-c5f1-4f26-8eef-d7ded353a325 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:51.160268 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a94f3621-c5f1-4f26-8eef-d7ded353a325 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:51.164118 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c33adae0-f46f-4cc8-b900-181de99d1341 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:51.164726 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c33adae0-f46f-4cc8-b900-181de99d1341 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:51.165005 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c33adae0-f46f-4cc8-b900-181de99d1341 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:51.165274 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c33adae0-f46f-4cc8-b900-181de99d1341 None admin] PATH_INFO: `/v3/services` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:51.165869 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c33adae0-f46f-4cc8-b900-181de99d1341 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:51.167226 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c33adae0-f46f-4cc8-b900-181de99d1341 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:51.173259 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 499/977] 10.222.0.33 () {56 vars in 1087 bytes} [Sun Oct 9 00:43:51 2022] GET /identity/v3/services => generated 1497 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:43:51.174021 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c33adae0-f46f-4cc8-b900-181de99d1341 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:51.300319 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c33adae0-f46f-4cc8-b900-181de99d1341 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:51.304613 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 479/978] 10.222.0.33 () {58 vars in 1282 bytes} [Sun Oct 9 00:43:50 2022] GET /identity/v3/auth/tokens => generated 1474 bytes in 380 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 09 00:43:51.309683 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a94f3621-c5f1-4f26-8eef-d7ded353a325 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:51.349393 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a957b015-9d77-478d-8d83-bb022d2abff9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:51.349925 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a957b015-9d77-478d-8d83-bb022d2abff9 None admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:51.350304 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a957b015-9d77-478d-8d83-bb022d2abff9 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:51.350585 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a957b015-9d77-478d-8d83-bb022d2abff9 None admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:51.351237 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a957b015-9d77-478d-8d83-bb022d2abff9 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:51.353624 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a957b015-9d77-478d-8d83-bb022d2abff9 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:51.366646 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:51.372075 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.sql.core [None req-a957b015-9d77-478d-8d83-bb022d2abff9 None admin] Conflict user: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Oct 09 00:43:51.372075 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Oct 09 00:43:51.372075 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [parameters: {'user_id': 'e2ab32c66091456e82d333cf3000d951', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Oct 09 00:43:51.372075 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81452) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 09 00:43:51.380016 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-a957b015-9d77-478d-8d83-bb022d2abff9 None admin] Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default.: keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Oct 09 00:43:51.380016 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:51.380016 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 09 00:43:51.380016 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 09 00:43:51.380016 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 09 00:43:51.380016 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 09 00:43:51.380016 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 09 00:43:51.380016 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application result = self._query(query) Oct 09 00:43:51.380016 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 09 00:43:51.380016 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application conn.query(q) Oct 09 00:43:51.380016 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 09 00:43:51.380016 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 09 00:43:51.380016 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 09 00:43:51.380016 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application result.read() Oct 09 00:43:51.380016 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 09 00:43:51.380879 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 09 00:43:51.380879 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 09 00:43:51.380879 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 09 00:43:51.380879 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 09 00:43:51.380879 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 09 00:43:51.380879 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 09 00:43:51.380879 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 09 00:43:51.380879 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Oct 09 00:43:51.380879 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:51.380879 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 09 00:43:51.380879 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:51.380879 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:51.380879 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 09 00:43:51.380879 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 09 00:43:51.380879 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 141, in create_user Oct 09 00:43:51.380879 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return base.filter_user(user_ref.to_dict()) Oct 09 00:43:51.380879 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 09 00:43:51.380879 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application next(self.gen) Oct 09 00:43:51.382389 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Oct 09 00:43:51.382389 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application yield resource Oct 09 00:43:51.382389 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 09 00:43:51.382389 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application next(self.gen) Oct 09 00:43:51.382389 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Oct 09 00:43:51.382389 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self.session.rollback() Oct 09 00:43:51.382389 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 09 00:43:51.382389 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self.force_reraise() Oct 09 00:43:51.382389 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 09 00:43:51.382389 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise self.value Oct 09 00:43:51.382389 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Oct 09 00:43:51.382389 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 09 00:43:51.382389 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Oct 09 00:43:51.382389 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application session.commit() Oct 09 00:43:51.382389 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 09 00:43:51.382389 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 09 00:43:51.382389 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 09 00:43:51.383269 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self._prepare_impl() Oct 09 00:43:51.383269 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 09 00:43:51.383269 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self.session.flush() Oct 09 00:43:51.383269 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 09 00:43:51.383269 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self._flush(objects) Oct 09 00:43:51.383269 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 09 00:43:51.383269 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 09 00:43:51.383269 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 09 00:43:51.383269 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application compat.raise_( Oct 09 00:43:51.383269 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 09 00:43:51.383269 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception Oct 09 00:43:51.383269 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 09 00:43:51.383269 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application flush_context.execute() Oct 09 00:43:51.383269 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 09 00:43:51.383269 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rec.execute(self) Oct 09 00:43:51.383269 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 09 00:43:51.383269 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 09 00:43:51.385541 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 09 00:43:51.385541 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 09 00:43:51.385541 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1238, in _emit_insert_statements Oct 09 00:43:51.385541 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application result = connection._execute_20( Oct 09 00:43:51.385541 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 09 00:43:51.385541 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 09 00:43:51.385541 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 09 00:43:51.385541 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 09 00:43:51.385541 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 09 00:43:51.385541 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 09 00:43:51.385541 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 09 00:43:51.385541 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 09 00:43:51.385541 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 09 00:43:51.385541 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 09 00:43:51.385541 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 09 00:43:51.385541 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception Oct 09 00:43:51.387395 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 09 00:43:51.387395 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 09 00:43:51.387395 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 09 00:43:51.387395 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 09 00:43:51.387395 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 09 00:43:51.387395 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application result = self._query(query) Oct 09 00:43:51.387395 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 09 00:43:51.387395 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application conn.query(q) Oct 09 00:43:51.387395 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 09 00:43:51.387395 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 09 00:43:51.387395 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 09 00:43:51.387395 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application result.read() Oct 09 00:43:51.387395 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 09 00:43:51.387395 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 09 00:43:51.387395 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 09 00:43:51.387395 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 09 00:43:51.389335 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 09 00:43:51.389335 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 09 00:43:51.389335 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 09 00:43:51.389335 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 09 00:43:51.389335 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Oct 09 00:43:51.389335 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Oct 09 00:43:51.389335 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [parameters: {'user_id': 'e2ab32c66091456e82d333cf3000d951', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Oct 09 00:43:51.389335 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 09 00:43:51.389335 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:51.389335 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 09 00:43:51.389335 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:51.389335 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:51.389335 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:51.389335 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:51.389335 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:51.390695 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:51.390695 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:51.390695 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:51.390695 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:51.390695 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:51.390695 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:51.390695 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:51.390695 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 244, in post Oct 09 00:43:51.390695 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.create_user( Oct 09 00:43:51.390695 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:51.390695 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:51.390695 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 09 00:43:51.390695 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 09 00:43:51.390695 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 09 00:43:51.390695 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 09 00:43:51.390695 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1000, in create_user Oct 09 00:43:51.390695 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ref = self._create_user_with_federated_objects(user, driver) Oct 09 00:43:51.394285 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 966, in _create_user_with_federated_objects Oct 09 00:43:51.394285 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application user = driver.create_user(user['id'], user) Oct 09 00:43:51.394285 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 09 00:43:51.394285 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 09 00:43:51.394285 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Oct 09 00:43:51.394285 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:51.394285 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 500/979] 10.222.0.33 () {60 vars in 1131 bytes} [Sun Oct 9 00:43:51 2022] POST /identity/v3/users => generated 160 bytes in 207 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:51.394285 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a957b015-9d77-478d-8d83-bb022d2abff9 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:51.402209 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:51.432058 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a94f3621-c5f1-4f26-8eef-d7ded353a325 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:51.471228 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4b9da28-4269-4cee-9d68-17b034f29ae4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:51.472133 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4b9da28-4269-4cee-9d68-17b034f29ae4 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:51.472331 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4b9da28-4269-4cee-9d68-17b034f29ae4 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:51.472509 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4b9da28-4269-4cee-9d68-17b034f29ae4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:51.483223 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4b9da28-4269-4cee-9d68-17b034f29ae4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:51.484730 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4b9da28-4269-4cee-9d68-17b034f29ae4 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:51.499751 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-a4b9da28-4269-4cee-9d68-17b034f29ae4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:51.499751 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:51.499751 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:51.499751 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:51.499751 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:51.499751 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:51.499751 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:51.499751 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:51.499751 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:51.499751 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:51.499751 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:51.499751 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:51.499751 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:43:51.499751 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:43:51.499751 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:43:51.499751 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:43:51.500745 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:51.500745 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:51.500745 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:51.500745 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:51.500745 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:51.500745 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:51.500745 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:51.500745 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:51.500745 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:51.500745 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:51.500745 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:51.500745 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:51.500745 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:51.500745 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:51.500745 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:51.500745 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:51.500745 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:43:51.501618 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:43:51.501618 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:43:51.501618 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:51.505564 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 480/980] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:43:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 09 00:43:51.507483 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4b9da28-4269-4cee-9d68-17b034f29ae4 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:51.530249 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a957b015-9d77-478d-8d83-bb022d2abff9 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:51.566996 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fc23280-a3ee-4b41-bf04-eb99287ead0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:51.567652 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fc23280-a3ee-4b41-bf04-eb99287ead0d None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:51.567934 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fc23280-a3ee-4b41-bf04-eb99287ead0d None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:51.568206 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fc23280-a3ee-4b41-bf04-eb99287ead0d None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:51.576837 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fc23280-a3ee-4b41-bf04-eb99287ead0d None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:51.578478 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fc23280-a3ee-4b41-bf04-eb99287ead0d None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:51.590775 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-3fc23280-a3ee-4b41-bf04-eb99287ead0d None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Oct 09 00:43:51.590775 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:51.590775 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:51.590775 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:51.590775 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:51.590775 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:51.590775 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:51.590775 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:51.590775 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:51.590775 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:51.590775 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:51.590775 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:51.590775 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 09 00:43:51.590775 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 09 00:43:51.590775 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 09 00:43:51.590775 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 09 00:43:51.591983 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:51.591983 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:51.591983 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 09 00:43:51.591983 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 09 00:43:51.591983 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 09 00:43:51.591983 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 09 00:43:51.591983 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:51.591983 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:51.591983 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:51.591983 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:51.591983 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:51.591983 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:51.591983 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:51.591983 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:51.591983 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:51.591983 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:51.591983 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:51.591983 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:51.592956 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:51.592956 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:51.592956 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 09 00:43:51.592956 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 09 00:43:51.592956 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 09 00:43:51.592956 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 09 00:43:51.592956 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 09 00:43:51.592956 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 09 00:43:51.592956 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Oct 09 00:43:51.592956 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:51.592956 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 501/981] 10.222.0.33 () {56 vars in 1099 bytes} [Sun Oct 9 00:43:51 2022] GET /identity/v3/users/cinder => generated 84 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:51.593557 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a6aefff-977e-4eb4-a841-fab0b4ed2eb8 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:51.594027 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a6aefff-977e-4eb4-a841-fab0b4ed2eb8 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:51.594027 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a6aefff-977e-4eb4-a841-fab0b4ed2eb8 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:51.598079 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 502/982] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:43:51 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:43:51.598789 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a6aefff-977e-4eb4-a841-fab0b4ed2eb8 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:51.626527 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4b9da28-4269-4cee-9d68-17b034f29ae4 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:51.628286 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:51.666333 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92e1acb1-5c02-48a0-a4e7-b4fabd912dd7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:51.666940 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e1acb1-5c02-48a0-a4e7-b4fabd912dd7 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:51.667221 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e1acb1-5c02-48a0-a4e7-b4fabd912dd7 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:51.667497 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e1acb1-5c02-48a0-a4e7-b4fabd912dd7 None admin] PATH_INFO: `/v3/domains` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:51.668163 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92e1acb1-5c02-48a0-a4e7-b4fabd912dd7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:51.668466 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92e1acb1-5c02-48a0-a4e7-b4fabd912dd7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:51.669826 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92e1acb1-5c02-48a0-a4e7-b4fabd912dd7 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:51.672901 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:51.687079 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 481/983] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:43:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:43:51.687079 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d61f8b-3f5b-49d9-bff2-dc6a53a2493f None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:51.687079 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d61f8b-3f5b-49d9-bff2-dc6a53a2493f None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:51.687079 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d61f8b-3f5b-49d9-bff2-dc6a53a2493f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:51.704422 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-a4d61f8b-3f5b-49d9-bff2-dc6a53a2493f None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:43:51.705608 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-a4d61f8b-3f5b-49d9-bff2-dc6a53a2493f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:43:51.718477 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a6aefff-977e-4eb4-a841-fab0b4ed2eb8 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:51.755854 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6bfac2f-a5b3-48e7-b5f6-05c3680a95c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:51.756471 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6bfac2f-a5b3-48e7-b5f6-05c3680a95c8 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:51.757294 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6bfac2f-a5b3-48e7-b5f6-05c3680a95c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:51.757570 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6bfac2f-a5b3-48e7-b5f6-05c3680a95c8 None admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:51.758666 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6bfac2f-a5b3-48e7-b5f6-05c3680a95c8 None admin] RBAC: Adding query filter params (name=cinder) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:51.758974 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6bfac2f-a5b3-48e7-b5f6-05c3680a95c8 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:51.760453 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6bfac2f-a5b3-48e7-b5f6-05c3680a95c8 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:51.780410 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 482/984] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:43:51 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 09 00:43:51.781528 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4d61f8b-3f5b-49d9-bff2-dc6a53a2493f None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:51.803340 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 503/985] 10.222.0.33 () {56 vars in 1101 bytes} [Sun Oct 9 00:43:51 2022] GET /identity/v3/users?name=cinder => generated 354 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:51.899771 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:51.909247 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4d61f8b-3f5b-49d9-bff2-dc6a53a2493f None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:51.947304 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5143e38d-669b-49a0-8872-e988326a17da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:51.947986 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5143e38d-669b-49a0-8872-e988326a17da None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:51.948265 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5143e38d-669b-49a0-8872-e988326a17da None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:51.948534 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5143e38d-669b-49a0-8872-e988326a17da None admin] PATH_INFO: `/v3/projects/service` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:51.956169 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5143e38d-669b-49a0-8872-e988326a17da None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:51.957632 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5143e38d-669b-49a0-8872-e988326a17da None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:51.970245 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-5143e38d-669b-49a0-8872-e988326a17da None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 09 00:43:51.970245 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:51.970245 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:51.970245 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:51.970245 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:51.970245 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:51.970245 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:51.970245 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:51.970245 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:51.970245 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:51.970245 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:51.970245 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:51.970245 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 09 00:43:51.970245 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 09 00:43:51.970245 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 09 00:43:51.971472 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 09 00:43:51.971472 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:51.971472 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:51.971472 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:51.971472 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:51.971472 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:51.971472 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:51.971472 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:51.971472 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:51.971472 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:51.971472 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:51.971472 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:51.971472 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:51.971472 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:51.971472 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:51.971472 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:51.971472 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:51.972785 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 09 00:43:51.972785 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 09 00:43:51.972785 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 09 00:43:51.972785 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 09 00:43:51.972785 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 09 00:43:51.972785 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 09 00:43:51.972785 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 09 00:43:51.972785 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:51.972785 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 483/986] 10.222.0.33 () {56 vars in 1111 bytes} [Sun Oct 9 00:43:51 2022] GET /identity/v3/projects/service => generated 88 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:51.977836 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6bfac2f-a5b3-48e7-b5f6-05c3680a95c8 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:52.088927 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5143e38d-669b-49a0-8872-e988326a17da None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:52.105224 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6bfac2f-a5b3-48e7-b5f6-05c3680a95c8 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:52.138274 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:52.150983 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4adb6371-e9b2-4a60-90c9-9ee523250e79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:52.151512 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4adb6371-e9b2-4a60-90c9-9ee523250e79 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:52.151821 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4adb6371-e9b2-4a60-90c9-9ee523250e79 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:52.152149 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4adb6371-e9b2-4a60-90c9-9ee523250e79 None admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:52.152795 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4adb6371-e9b2-4a60-90c9-9ee523250e79 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:52.153297 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4adb6371-e9b2-4a60-90c9-9ee523250e79 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:52.154521 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4adb6371-e9b2-4a60-90c9-9ee523250e79 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:52.167914 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 504/987] 10.222.0.33 () {56 vars in 1148 bytes} [Sun Oct 9 00:43:51 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:52.173793 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4adb6371-e9b2-4a60-90c9-9ee523250e79 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:52.209519 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5143e38d-669b-49a0-8872-e988326a17da None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:52.245184 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87114957-3534-4c81-aa60-a84880ddc2bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:52.245870 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87114957-3534-4c81-aa60-a84880ddc2bf None admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:52.246076 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87114957-3534-4c81-aa60-a84880ddc2bf None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:52.246348 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87114957-3534-4c81-aa60-a84880ddc2bf None admin] PATH_INFO: `/v3/services` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:52.246990 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87114957-3534-4c81-aa60-a84880ddc2bf None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:52.248384 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87114957-3534-4c81-aa60-a84880ddc2bf None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:52.263988 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 484/988] 10.222.0.33 () {60 vars in 1141 bytes} [Sun Oct 9 00:43:52 2022] POST /identity/v3/services => generated 242 bytes in 176 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 09 00:43:52.282510 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4adb6371-e9b2-4a60-90c9-9ee523250e79 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:52.330340 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3096bf7b-f508-46bc-a812-908c7057ce8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:52.331398 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3096bf7b-f508-46bc-a812-908c7057ce8e None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:52.331897 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3096bf7b-f508-46bc-a812-908c7057ce8e None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:52.332371 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3096bf7b-f508-46bc-a812-908c7057ce8e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:52.333285 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3096bf7b-f508-46bc-a812-908c7057ce8e None admin] RBAC: Adding query filter params (role.id=fddb7718710e48bb8034bd9390b820a2, scope.project.id=013c83476b1340fbb051061833c4a9c6, user.id=5f3247ddcda54394995d844c776bf43c) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:52.333933 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3096bf7b-f508-46bc-a812-908c7057ce8e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:52.335640 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3096bf7b-f508-46bc-a812-908c7057ce8e None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:52.343158 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 505/989] 10.222.0.33 () {56 vars in 1374 bytes} [Sun Oct 9 00:43:52 2022] GET /identity/v3/role_assignments?user.id=5f3247ddcda54394995d844c776bf43c&scope.project.id=013c83476b1340fbb051061833c4a9c6&role.id=fddb7718710e48bb8034bd9390b820a2 => generated 608 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:53.421986 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a1a6b09-3477-4720-a646-ed9ae000193e None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:53.422516 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a1a6b09-3477-4720-a646-ed9ae000193e None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:53.422516 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a1a6b09-3477-4720-a646-ed9ae000193e None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:53.423929 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 485/990] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:43:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:43:53.429558 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff578c43-ba01-4764-907d-d6187a366c84 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:53.429975 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff578c43-ba01-4764-907d-d6187a366c84 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:53.430118 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff578c43-ba01-4764-907d-d6187a366c84 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:53.449081 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-ff578c43-ba01-4764-907d-d6187a366c84 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:43:53.450576 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-ff578c43-ba01-4764-907d-d6187a366c84 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:43:53.493130 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:53.518974 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 506/991] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:43:53 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 09 00:43:53.783953 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a1a6b09-3477-4720-a646-ed9ae000193e None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:53.799965 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:53.887416 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a1a6b09-3477-4720-a646-ed9ae000193e None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:53.920434 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c56c8c36-dab9-458f-a817-99e8bb528616 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:53.921062 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c56c8c36-dab9-458f-a817-99e8bb528616 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:53.921341 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c56c8c36-dab9-458f-a817-99e8bb528616 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:53.921614 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c56c8c36-dab9-458f-a817-99e8bb528616 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:53.922197 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c56c8c36-dab9-458f-a817-99e8bb528616 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:53.923554 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c56c8c36-dab9-458f-a817-99e8bb528616 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:53.933056 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-c56c8c36-dab9-458f-a817-99e8bb528616 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 09 00:43:53.933056 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:53.933056 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 09 00:43:53.933056 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 09 00:43:53.933056 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 09 00:43:53.933056 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 09 00:43:53.933056 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 09 00:43:53.933056 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 09 00:43:53.933056 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 09 00:43:53.933056 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:53.933056 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 09 00:43:53.933056 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:53.933056 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:53.933056 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:53.933056 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:53.933056 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:53.933056 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:53.934058 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:53.934058 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:53.934058 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:53.934058 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:53.934058 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:53.934058 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:53.934058 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 09 00:43:53.934058 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 09 00:43:53.934058 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 09 00:43:53.934058 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 09 00:43:53.934058 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:53.934058 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:53.934058 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:53.934058 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:53.934058 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:53.934058 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:53.934964 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:53.934964 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:53.934964 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:53.934964 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:53.934964 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:53.934964 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:53.934964 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:53.934964 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:53.934964 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:53.934964 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:53.934964 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 09 00:43:53.934964 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 09 00:43:53.934964 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 09 00:43:53.934964 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:53.939149 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 486/992] 10.222.0.33 () {56 vars in 1126 bytes} [Sun Oct 9 00:43:53 2022] GET /identity/v3/services/object-store => generated 93 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:53.939999 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff578c43-ba01-4764-907d-d6187a366c84 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:54.040030 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff578c43-ba01-4764-907d-d6187a366c84 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:54.074049 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-906673dc-5c11-4305-9ade-2be069af2ad9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:54.074656 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-906673dc-5c11-4305-9ade-2be069af2ad9 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:54.074932 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-906673dc-5c11-4305-9ade-2be069af2ad9 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:54.075641 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-906673dc-5c11-4305-9ade-2be069af2ad9 None admin] PATH_INFO: `/v3/services` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:54.077338 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-906673dc-5c11-4305-9ade-2be069af2ad9 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:54.077338 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-906673dc-5c11-4305-9ade-2be069af2ad9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:54.077527 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-906673dc-5c11-4305-9ade-2be069af2ad9 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:54.083610 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 507/993] 10.222.0.33 () {56 vars in 1122 bytes} [Sun Oct 9 00:43:53 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:54.090360 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c56c8c36-dab9-458f-a817-99e8bb528616 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:54.168972 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:54.193213 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c56c8c36-dab9-458f-a817-99e8bb528616 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:54.226879 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c84d1a3-3863-4bad-a90d-ac8a21231275 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:54.227479 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c84d1a3-3863-4bad-a90d-ac8a21231275 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:54.227787 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c84d1a3-3863-4bad-a90d-ac8a21231275 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:54.228061 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c84d1a3-3863-4bad-a90d-ac8a21231275 None admin] PATH_INFO: `/v3/services` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:54.228689 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c84d1a3-3863-4bad-a90d-ac8a21231275 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:54.228990 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c84d1a3-3863-4bad-a90d-ac8a21231275 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:54.230346 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c84d1a3-3863-4bad-a90d-ac8a21231275 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:54.235212 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 487/994] 10.222.0.33 () {56 vars in 1122 bytes} [Sun Oct 9 00:43:54 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:54.240664 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-906673dc-5c11-4305-9ade-2be069af2ad9 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:54.283287 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:54.343422 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-906673dc-5c11-4305-9ade-2be069af2ad9 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:54.381586 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9749c2c-1e17-4c0b-910c-e9e14d1e8f1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:54.382720 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9749c2c-1e17-4c0b-910c-e9e14d1e8f1e None admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:54.382999 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9749c2c-1e17-4c0b-910c-e9e14d1e8f1e None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:54.383271 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9749c2c-1e17-4c0b-910c-e9e14d1e8f1e None admin] PATH_INFO: `/v3/endpoints` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:54.383974 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9749c2c-1e17-4c0b-910c-e9e14d1e8f1e None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:54.385335 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9749c2c-1e17-4c0b-910c-e9e14d1e8f1e None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:54.402546 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 508/995] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:43:54 2022] POST /identity/v3/endpoints => generated 325 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:43:54.463846 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62bca41d-4343-4d8e-804f-4634080a8d6f None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:54.464382 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62bca41d-4343-4d8e-804f-4634080a8d6f None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:54.464382 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62bca41d-4343-4d8e-804f-4634080a8d6f None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:54.465965 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 488/996] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:43:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:43:54.471749 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a6748e6-01c5-4704-be77-f537f45ad3ea None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:54.472252 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a6748e6-01c5-4704-be77-f537f45ad3ea None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:54.472325 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a6748e6-01c5-4704-be77-f537f45ad3ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:54.492445 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-4a6748e6-01c5-4704-be77-f537f45ad3ea None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:43:54.493666 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-4a6748e6-01c5-4704-be77-f537f45ad3ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:43:54.559163 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ddffab-0481-45b2-a644-223eac03ecc5 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:54.559441 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ddffab-0481-45b2-a644-223eac03ecc5 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:54.559752 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7ddffab-0481-45b2-a644-223eac03ecc5 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:54.561085 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 489/997] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:43:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 09 00:43:54.566887 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7284599f-bd6f-4c57-94d3-ed5f65a9a013 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:54.567178 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7284599f-bd6f-4c57-94d3-ed5f65a9a013 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:54.567448 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7284599f-bd6f-4c57-94d3-ed5f65a9a013 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:54.578088 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 509/998] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:43:54 2022] POST /identity/v3/auth/tokens => generated 1717 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 09 00:43:54.585868 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-7284599f-bd6f-4c57-94d3-ed5f65a9a013 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:43:54.587074 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-7284599f-bd6f-4c57-94d3-ed5f65a9a013 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:43:54.657921 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 490/999] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:43:54 2022] POST /identity/v3/auth/tokens => generated 1717 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 09 00:43:54.848021 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a6748e6-01c5-4704-be77-f537f45ad3ea None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:54.875403 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:54.934496 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7284599f-bd6f-4c57-94d3-ed5f65a9a013 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:54.952669 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a6748e6-01c5-4704-be77-f537f45ad3ea None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:54.989075 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d8f7e00-50de-4d8e-ba07-f5ae34ac5a28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:54.989704 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d8f7e00-50de-4d8e-ba07-f5ae34ac5a28 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:54.989984 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d8f7e00-50de-4d8e-ba07-f5ae34ac5a28 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:54.990257 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d8f7e00-50de-4d8e-ba07-f5ae34ac5a28 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:54.990839 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d8f7e00-50de-4d8e-ba07-f5ae34ac5a28 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:54.992337 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d8f7e00-50de-4d8e-ba07-f5ae34ac5a28 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:55.001024 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-3d8f7e00-50de-4d8e-ba07-f5ae34ac5a28 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Oct 09 00:43:55.001024 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:55.001024 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 09 00:43:55.001024 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 09 00:43:55.001024 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 09 00:43:55.001024 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 09 00:43:55.001024 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 09 00:43:55.001024 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 09 00:43:55.001024 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 09 00:43:55.001024 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:55.001024 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 09 00:43:55.001024 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:55.001024 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:55.001024 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:55.001024 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:55.001024 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:55.001024 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:55.001991 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:55.001991 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:55.001991 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:55.001991 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:55.001991 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:55.001991 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:55.001991 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 09 00:43:55.001991 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 09 00:43:55.001991 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 09 00:43:55.001991 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 09 00:43:55.001991 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:55.001991 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:55.001991 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:55.001991 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:55.001991 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:55.001991 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:55.002828 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:55.002828 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:55.002828 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:55.002828 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:55.002828 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:55.002828 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:55.002828 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:55.002828 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:55.002828 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:55.002828 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:55.002828 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 09 00:43:55.002828 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 09 00:43:55.002828 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 09 00:43:55.002828 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:43:55.002828 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 510/1000] 10.222.0.33 () {56 vars in 1111 bytes} [Sun Oct 9 00:43:54 2022] GET /identity/v3/services/compute => generated 88 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:55.007944 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d8f7e00-50de-4d8e-ba07-f5ae34ac5a28 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:55.012396 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:55.045352 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7284599f-bd6f-4c57-94d3-ed5f65a9a013 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:55.088169 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-265ee224-e8b5-4f1f-9e55-14624872fe54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:55.088809 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-265ee224-e8b5-4f1f-9e55-14624872fe54 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:55.089089 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-265ee224-e8b5-4f1f-9e55-14624872fe54 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:55.089364 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-265ee224-e8b5-4f1f-9e55-14624872fe54 None admin] PATH_INFO: `/v3/services/image` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:55.089942 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-265ee224-e8b5-4f1f-9e55-14624872fe54 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:55.091302 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-265ee224-e8b5-4f1f-9e55-14624872fe54 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:55.100086 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-265ee224-e8b5-4f1f-9e55-14624872fe54 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Oct 09 00:43:55.100086 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:55.100086 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 09 00:43:55.100086 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 09 00:43:55.100086 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 09 00:43:55.100086 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 09 00:43:55.100086 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 09 00:43:55.100086 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 09 00:43:55.100086 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 09 00:43:55.100086 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:55.100086 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 09 00:43:55.100086 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:55.100086 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:55.100086 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:55.100086 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:55.100086 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:55.100086 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:55.101696 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:55.101696 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:55.101696 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:55.101696 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:55.101696 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:55.101696 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:55.101696 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 09 00:43:55.101696 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 09 00:43:55.101696 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 09 00:43:55.101696 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 09 00:43:55.101696 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:55.101696 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:55.101696 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:55.101696 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:55.101696 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:55.101696 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:55.103368 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:55.103368 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:55.103368 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:55.103368 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:55.103368 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:55.103368 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:55.103368 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:55.103368 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:55.103368 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:55.103368 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:55.103368 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 09 00:43:55.103368 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 09 00:43:55.103368 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 09 00:43:55.103368 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:55.103368 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 491/1001] 10.222.0.33 () {56 vars in 1105 bytes} [Sun Oct 9 00:43:54 2022] GET /identity/v3/services/image => generated 86 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:55.107647 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-265ee224-e8b5-4f1f-9e55-14624872fe54 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:55.114290 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:55.126026 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d8f7e00-50de-4d8e-ba07-f5ae34ac5a28 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:55.165256 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f27e50e-84ce-40cd-8cd3-f076a923749b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:55.166210 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f27e50e-84ce-40cd-8cd3-f076a923749b None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:55.166652 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f27e50e-84ce-40cd-8cd3-f076a923749b None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:55.166929 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f27e50e-84ce-40cd-8cd3-f076a923749b None admin] PATH_INFO: `/v3/services` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:55.167561 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f27e50e-84ce-40cd-8cd3-f076a923749b None admin] RBAC: Adding query filter params (name=compute) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:55.167898 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f27e50e-84ce-40cd-8cd3-f076a923749b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:55.169278 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f27e50e-84ce-40cd-8cd3-f076a923749b None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:55.174824 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 511/1002] 10.222.0.33 () {56 vars in 1112 bytes} [Sun Oct 9 00:43:55 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:43:55.180938 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f27e50e-84ce-40cd-8cd3-f076a923749b None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:55.220635 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-265ee224-e8b5-4f1f-9e55-14624872fe54 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:55.256400 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0511d5d-a075-4cf7-afae-bd7f6448c88c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:55.257782 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0511d5d-a075-4cf7-afae-bd7f6448c88c None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:55.258197 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0511d5d-a075-4cf7-afae-bd7f6448c88c None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:55.258529 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0511d5d-a075-4cf7-afae-bd7f6448c88c None admin] PATH_INFO: `/v3/services` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:55.259272 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0511d5d-a075-4cf7-afae-bd7f6448c88c None admin] RBAC: Adding query filter params (name=image) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:55.259575 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0511d5d-a075-4cf7-afae-bd7f6448c88c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:55.260975 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0511d5d-a075-4cf7-afae-bd7f6448c88c None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:55.262140 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:55.272156 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 492/1003] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:43:55 2022] GET /identity/v3/services?name=image => generated 122 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:43:55.274253 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0511d5d-a075-4cf7-afae-bd7f6448c88c None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:55.284303 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f27e50e-84ce-40cd-8cd3-f076a923749b None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:55.320129 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9086efe1-b081-4c8a-b881-19b431dc9f5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:55.321393 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9086efe1-b081-4c8a-b881-19b431dc9f5b None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:55.321678 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9086efe1-b081-4c8a-b881-19b431dc9f5b None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:55.321956 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9086efe1-b081-4c8a-b881-19b431dc9f5b None admin] PATH_INFO: `/v3/services` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:55.322614 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9086efe1-b081-4c8a-b881-19b431dc9f5b None admin] RBAC: Adding query filter params (type=compute) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:55.323335 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9086efe1-b081-4c8a-b881-19b431dc9f5b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:55.325189 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9086efe1-b081-4c8a-b881-19b431dc9f5b None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:55.331210 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 512/1004] 10.222.0.33 () {56 vars in 1112 bytes} [Sun Oct 9 00:43:55 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:43:55.337526 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9086efe1-b081-4c8a-b881-19b431dc9f5b None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:55.361012 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:55.388971 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0511d5d-a075-4cf7-afae-bd7f6448c88c None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:55.425638 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71f8c3e2-c018-4ba2-b858-4701aa34d316 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:55.426128 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71f8c3e2-c018-4ba2-b858-4701aa34d316 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:55.426415 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71f8c3e2-c018-4ba2-b858-4701aa34d316 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:55.426776 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71f8c3e2-c018-4ba2-b858-4701aa34d316 None admin] PATH_INFO: `/v3/services` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:55.427301 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71f8c3e2-c018-4ba2-b858-4701aa34d316 None admin] RBAC: Adding query filter params (type=image) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:55.427625 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71f8c3e2-c018-4ba2-b858-4701aa34d316 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:55.429797 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71f8c3e2-c018-4ba2-b858-4701aa34d316 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:55.434706 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 493/1005] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:43:55 2022] GET /identity/v3/services?type=image => generated 122 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:55.445279 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9086efe1-b081-4c8a-b881-19b431dc9f5b None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:55.483534 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1e885d5-af33-40a7-afa5-ace1517ecdd8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:55.484227 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1e885d5-af33-40a7-afa5-ace1517ecdd8 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:55.484444 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1e885d5-af33-40a7-afa5-ace1517ecdd8 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:55.484638 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1e885d5-af33-40a7-afa5-ace1517ecdd8 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:55.485359 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1e885d5-af33-40a7-afa5-ace1517ecdd8 None admin] RBAC: Adding query filter params (interface=public, service_id=47f2732c94804529ab29c0de08ca160e, region_id=RegionOne) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:55.485672 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1e885d5-af33-40a7-afa5-ace1517ecdd8 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:55.487092 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1e885d5-af33-40a7-afa5-ace1517ecdd8 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:55.498624 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 513/1006] 10.222.0.33 () {56 vars in 1251 bytes} [Sun Oct 9 00:43:55 2022] GET /identity/v3/endpoints?service_id=47f2732c94804529ab29c0de08ca160e&interface=public®ion_id=RegionOne => generated 194 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:55.500366 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71f8c3e2-c018-4ba2-b858-4701aa34d316 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:55.570755 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:55.615505 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71f8c3e2-c018-4ba2-b858-4701aa34d316 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:55.652049 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7730ad8c-0e11-4ac6-b01b-7e6f92289e78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:55.652664 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7730ad8c-0e11-4ac6-b01b-7e6f92289e78 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:55.652943 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7730ad8c-0e11-4ac6-b01b-7e6f92289e78 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:55.653216 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7730ad8c-0e11-4ac6-b01b-7e6f92289e78 None admin] PATH_INFO: `/v3/services` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:55.653821 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7730ad8c-0e11-4ac6-b01b-7e6f92289e78 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:55.655175 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7730ad8c-0e11-4ac6-b01b-7e6f92289e78 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:55.667272 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 494/1007] 10.222.0.33 () {56 vars in 1087 bytes} [Sun Oct 9 00:43:55 2022] GET /identity/v3/services => generated 1727 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:43:56.603517 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-694f287d-b45d-4f97-9c14-6be2e96416f5 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:56.604068 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-694f287d-b45d-4f97-9c14-6be2e96416f5 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:56.604068 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-694f287d-b45d-4f97-9c14-6be2e96416f5 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:56.605457 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 514/1008] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:43:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:43:56.610628 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-408b98af-d886-4547-b5f2-a29da34845e0 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:56.611042 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-408b98af-d886-4547-b5f2-a29da34845e0 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:56.611183 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-408b98af-d886-4547-b5f2-a29da34845e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:56.629764 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-408b98af-d886-4547-b5f2-a29da34845e0 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:43:56.630936 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-408b98af-d886-4547-b5f2-a29da34845e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:43:56.698094 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 495/1009] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:43:56 2022] POST /identity/v3/auth/tokens => generated 1717 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 09 00:43:56.964170 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-694f287d-b45d-4f97-9c14-6be2e96416f5 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:57.050861 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:57.077614 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-694f287d-b45d-4f97-9c14-6be2e96416f5 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:57.112695 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55c2f18f-b9b3-4219-8e35-31dd03e20bca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:57.113311 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c2f18f-b9b3-4219-8e35-31dd03e20bca None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:57.113589 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c2f18f-b9b3-4219-8e35-31dd03e20bca None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:57.113872 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c2f18f-b9b3-4219-8e35-31dd03e20bca None admin] PATH_INFO: `/v3/domains/default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:57.114939 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55c2f18f-b9b3-4219-8e35-31dd03e20bca None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:57.116964 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55c2f18f-b9b3-4219-8e35-31dd03e20bca None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:57.118688 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 515/1010] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:43:56 2022] GET /identity/v3/domains/default => generated 199 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:57.124366 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-408b98af-d886-4547-b5f2-a29da34845e0 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:57.179360 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:57.224178 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-408b98af-d886-4547-b5f2-a29da34845e0 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:57.258148 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-722b69e8-35ec-4bb8-87fa-b305ae19a16c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:57.258741 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-722b69e8-35ec-4bb8-87fa-b305ae19a16c None admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:57.259016 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-722b69e8-35ec-4bb8-87fa-b305ae19a16c None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:57.259284 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-722b69e8-35ec-4bb8-87fa-b305ae19a16c None admin] PATH_INFO: `/v3/projects` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:57.259953 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-722b69e8-35ec-4bb8-87fa-b305ae19a16c None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:57.261359 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-722b69e8-35ec-4bb8-87fa-b305ae19a16c None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:57.291632 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 496/1011] 10.222.0.33 () {60 vars in 1141 bytes} [Sun Oct 9 00:43:57 2022] POST /identity/v3/projects => generated 311 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:43:57.588120 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b94382-c9c2-44cd-8a13-f2b917f2bcb9 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:57.588409 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b94382-c9c2-44cd-8a13-f2b917f2bcb9 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:57.588599 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b94382-c9c2-44cd-8a13-f2b917f2bcb9 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:57.590066 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 516/1012] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:43:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:43:57.595070 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ed0dd26-4632-4a41-b77c-63b18b946d65 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:57.595365 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ed0dd26-4632-4a41-b77c-63b18b946d65 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:57.595677 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ed0dd26-4632-4a41-b77c-63b18b946d65 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:57.614879 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-7ed0dd26-4632-4a41-b77c-63b18b946d65 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:43:57.616662 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-7ed0dd26-4632-4a41-b77c-63b18b946d65 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:43:57.683354 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 497/1013] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:43:57 2022] POST /identity/v3/auth/tokens => generated 1717 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 09 00:43:57.820569 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccfe3a5f-5491-4030-a01c-cf980ca67d7a None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:57.821074 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccfe3a5f-5491-4030-a01c-cf980ca67d7a None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:57.821152 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccfe3a5f-5491-4030-a01c-cf980ca67d7a None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:57.822580 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 517/1014] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:43:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:43:57.827565 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a271d1-5a0f-42ff-a76f-97b285b7b8f0 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:57.827885 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a271d1-5a0f-42ff-a76f-97b285b7b8f0 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:57.828157 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a271d1-5a0f-42ff-a76f-97b285b7b8f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:57.846665 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-d9a271d1-5a0f-42ff-a76f-97b285b7b8f0 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:43:57.847799 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-d9a271d1-5a0f-42ff-a76f-97b285b7b8f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:43:57.877273 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:57.917495 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 498/1015] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:43:57 2022] POST /identity/v3/auth/tokens => generated 1717 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 09 00:43:57.949041 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccfe3a5f-5491-4030-a01c-cf980ca67d7a None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:58.051892 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccfe3a5f-5491-4030-a01c-cf980ca67d7a None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:58.093307 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-922304a7-37fb-489d-8d51-395b766aa83d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:58.093912 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-922304a7-37fb-489d-8d51-395b766aa83d None admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:58.094189 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-922304a7-37fb-489d-8d51-395b766aa83d None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:58.094459 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-922304a7-37fb-489d-8d51-395b766aa83d None admin] PATH_INFO: `/v3/services` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:58.095108 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-922304a7-37fb-489d-8d51-395b766aa83d None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:58.096504 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-922304a7-37fb-489d-8d51-395b766aa83d None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:58.113333 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 518/1016] 10.222.0.33 () {60 vars in 1141 bytes} [Sun Oct 9 00:43:57 2022] POST /identity/v3/services => generated 242 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:43:58.184086 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9a271d1-5a0f-42ff-a76f-97b285b7b8f0 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:58.292505 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9a271d1-5a0f-42ff-a76f-97b285b7b8f0 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:58.333518 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb3ea1bb-81d7-4692-98bf-80929a95331c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:58.334129 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3ea1bb-81d7-4692-98bf-80929a95331c None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:58.334402 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3ea1bb-81d7-4692-98bf-80929a95331c None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:58.334708 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3ea1bb-81d7-4692-98bf-80929a95331c None admin] PATH_INFO: `/v3/services/compute` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:58.335275 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb3ea1bb-81d7-4692-98bf-80929a95331c None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:58.336688 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb3ea1bb-81d7-4692-98bf-80929a95331c None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:58.345292 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-eb3ea1bb-81d7-4692-98bf-80929a95331c None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Oct 09 00:43:58.345292 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:58.345292 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 09 00:43:58.345292 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 09 00:43:58.345292 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 09 00:43:58.345292 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 09 00:43:58.345292 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 09 00:43:58.345292 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 09 00:43:58.345292 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 09 00:43:58.345292 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:58.345292 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 09 00:43:58.345292 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:58.345292 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:43:58.345292 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:43:58.345292 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:43:58.345292 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:43:58.345292 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:43:58.346254 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:43:58.346254 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:43:58.346254 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:43:58.346254 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:43:58.346254 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:43:58.346254 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:43:58.346254 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 09 00:43:58.346254 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 09 00:43:58.346254 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 09 00:43:58.346254 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 09 00:43:58.346254 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:43:58.346254 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:43:58.346254 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:43:58.346254 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:43:58.346254 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:43:58.346254 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:43:58.347140 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:43:58.347140 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:43:58.347140 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:43:58.347140 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:43:58.347140 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:43:58.347140 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:43:58.347140 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:43:58.347140 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:43:58.347140 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:43:58.347140 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:43:58.347140 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 09 00:43:58.347140 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 09 00:43:58.347140 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 09 00:43:58.347140 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:43:58.347140 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 499/1017] 10.222.0.33 () {56 vars in 1111 bytes} [Sun Oct 9 00:43:58 2022] GET /identity/v3/services/compute => generated 88 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:43:58.352949 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-922304a7-37fb-489d-8d51-395b766aa83d None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:58.365690 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:58.455121 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-922304a7-37fb-489d-8d51-395b766aa83d None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:58.488973 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21301ed8-eda9-4eba-9fb6-1c0ac10b5709 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:58.489580 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21301ed8-eda9-4eba-9fb6-1c0ac10b5709 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:58.489862 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21301ed8-eda9-4eba-9fb6-1c0ac10b5709 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:58.490138 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21301ed8-eda9-4eba-9fb6-1c0ac10b5709 None admin] PATH_INFO: `/v3/services` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:58.490766 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21301ed8-eda9-4eba-9fb6-1c0ac10b5709 None admin] RBAC: Adding query filter params (name=compute) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:58.491100 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21301ed8-eda9-4eba-9fb6-1c0ac10b5709 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:58.492494 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21301ed8-eda9-4eba-9fb6-1c0ac10b5709 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:58.501934 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 519/1018] 10.222.0.33 () {56 vars in 1112 bytes} [Sun Oct 9 00:43:58 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:58.503048 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb3ea1bb-81d7-4692-98bf-80929a95331c None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:58.532149 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:58.604640 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb3ea1bb-81d7-4692-98bf-80929a95331c None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:58.639733 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a240b658-af31-4351-b1a2-5e6a29357856 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:58.640328 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a240b658-af31-4351-b1a2-5e6a29357856 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:58.640554 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a240b658-af31-4351-b1a2-5e6a29357856 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:58.640799 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a240b658-af31-4351-b1a2-5e6a29357856 None admin] PATH_INFO: `/v3/services` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:58.641422 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a240b658-af31-4351-b1a2-5e6a29357856 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:43:58.641727 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a240b658-af31-4351-b1a2-5e6a29357856 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:58.643083 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a240b658-af31-4351-b1a2-5e6a29357856 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:58.654070 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 500/1019] 10.222.0.33 () {56 vars in 1112 bytes} [Sun Oct 9 00:43:58 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:43:58.654229 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21301ed8-eda9-4eba-9fb6-1c0ac10b5709 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:58.737232 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:58.763758 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21301ed8-eda9-4eba-9fb6-1c0ac10b5709 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:43:58.797536 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52c9c955-2e87-40cf-878c-907caf804b6d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:43:58.798150 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c9c955-2e87-40cf-878c-907caf804b6d None admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:58.798432 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c9c955-2e87-40cf-878c-907caf804b6d None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:58.798708 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c9c955-2e87-40cf-878c-907caf804b6d None admin] PATH_INFO: `/v3/endpoints` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:58.799363 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52c9c955-2e87-40cf-878c-907caf804b6d None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:43:58.800759 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52c9c955-2e87-40cf-878c-907caf804b6d None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:43:58.820975 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 520/1020] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:43:58 2022] POST /identity/v3/endpoints => generated 334 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 09 00:43:59.510625 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-608de757-5da3-4ea5-a044-d598adf2922f None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:59.510948 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-608de757-5da3-4ea5-a044-d598adf2922f None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:59.511177 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-608de757-5da3-4ea5-a044-d598adf2922f None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:59.512572 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 501/1021] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:43:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:43:59.517623 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6625c8-6316-4249-a4fb-6ad0cbb571a8 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:43:59.517916 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6625c8-6316-4249-a4fb-6ad0cbb571a8 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:43:59.518193 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6625c8-6316-4249-a4fb-6ad0cbb571a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:43:59.535860 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-2e6625c8-6316-4249-a4fb-6ad0cbb571a8 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:43:59.537019 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-2e6625c8-6316-4249-a4fb-6ad0cbb571a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:43:59.609845 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 521/1022] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:43:59 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 09 00:43:59.881403 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-608de757-5da3-4ea5-a044-d598adf2922f None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:43:59.976262 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:43:59.987435 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-608de757-5da3-4ea5-a044-d598adf2922f None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:00.021711 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51bd15c2-7e74-47d1-b682-c36477ad1c73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:00.022843 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51bd15c2-7e74-47d1-b682-c36477ad1c73 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:00.023122 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51bd15c2-7e74-47d1-b682-c36477ad1c73 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:00.023392 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51bd15c2-7e74-47d1-b682-c36477ad1c73 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:00.024515 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51bd15c2-7e74-47d1-b682-c36477ad1c73 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:00.025930 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51bd15c2-7e74-47d1-b682-c36477ad1c73 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:00.027689 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 502/1023] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:43:59 2022] GET /identity/v3/domains/default => generated 199 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:44:00.033385 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e6625c8-6316-4249-a4fb-6ad0cbb571a8 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:00.096066 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:00.147838 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e6625c8-6316-4249-a4fb-6ad0cbb571a8 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:00.182893 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15ea1338-d12b-4e2f-88d4-5f82cea120d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:00.183495 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ea1338-d12b-4e2f-88d4-5f82cea120d8 None admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:00.183805 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ea1338-d12b-4e2f-88d4-5f82cea120d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:00.184078 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ea1338-d12b-4e2f-88d4-5f82cea120d8 None admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:00.184731 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15ea1338-d12b-4e2f-88d4-5f82cea120d8 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:00.186152 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15ea1338-d12b-4e2f-88d4-5f82cea120d8 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:00.211437 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 522/1024] 10.222.0.33 () {60 vars in 1132 bytes} [Sun Oct 9 00:44:00 2022] POST /identity/v3/users => generated 285 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:44:00.322433 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3fbe67-4952-4bb7-975e-843e06959766 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:00.322719 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3fbe67-4952-4bb7-975e-843e06959766 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:00.323000 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3fbe67-4952-4bb7-975e-843e06959766 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:00.324392 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 503/1025] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:44:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 09 00:44:00.330031 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a64484db-c68a-42d2-a5fe-4628511d46ca None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:00.330324 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a64484db-c68a-42d2-a5fe-4628511d46ca None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:00.330601 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a64484db-c68a-42d2-a5fe-4628511d46ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:00.349392 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-a64484db-c68a-42d2-a5fe-4628511d46ca None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:44:00.350684 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-a64484db-c68a-42d2-a5fe-4628511d46ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:44:00.425305 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 523/1026] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:44:00 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 09 00:44:00.695991 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca3fbe67-4952-4bb7-975e-843e06959766 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:00.801314 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca3fbe67-4952-4bb7-975e-843e06959766 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:00.835600 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55532629-0546-4c16-828b-3c8b5b72ee04 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:00.836260 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55532629-0546-4c16-828b-3c8b5b72ee04 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:00.836539 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55532629-0546-4c16-828b-3c8b5b72ee04 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:00.836811 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55532629-0546-4c16-828b-3c8b5b72ee04 None admin] PATH_INFO: `/v3/services/image` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:00.837385 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55532629-0546-4c16-828b-3c8b5b72ee04 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:00.839064 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55532629-0546-4c16-828b-3c8b5b72ee04 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:00.847725 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-55532629-0546-4c16-828b-3c8b5b72ee04 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Oct 09 00:44:00.847725 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:44:00.847725 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 09 00:44:00.847725 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 09 00:44:00.847725 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 09 00:44:00.847725 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 09 00:44:00.847725 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 09 00:44:00.847725 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 09 00:44:00.847725 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 09 00:44:00.847725 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:44:00.847725 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 09 00:44:00.847725 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:44:00.847725 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:44:00.847725 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:44:00.847725 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:44:00.847725 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:44:00.847725 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:44:00.848677 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:44:00.848677 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:44:00.848677 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:44:00.848677 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:44:00.848677 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:44:00.848677 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:44:00.848677 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 09 00:44:00.848677 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 09 00:44:00.848677 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 09 00:44:00.848677 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 09 00:44:00.848677 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:44:00.848677 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:44:00.848677 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:44:00.848677 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:44:00.848677 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:44:00.848677 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:44:00.849500 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:44:00.849500 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:44:00.849500 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:44:00.849500 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:44:00.849500 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:44:00.849500 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:44:00.849500 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:44:00.849500 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:44:00.849500 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:44:00.849500 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:44:00.849500 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 09 00:44:00.849500 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 09 00:44:00.849500 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 09 00:44:00.849500 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:44:00.851904 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 504/1027] 10.222.0.33 () {56 vars in 1105 bytes} [Sun Oct 9 00:44:00 2022] GET /identity/v3/services/image => generated 86 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:44:00.855486 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a64484db-c68a-42d2-a5fe-4628511d46ca None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:00.899181 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:00.959380 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a64484db-c68a-42d2-a5fe-4628511d46ca None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:00.998824 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ff708ed-e24c-4891-9818-0713262b0761 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:00.999453 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff708ed-e24c-4891-9818-0713262b0761 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:00.999773 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff708ed-e24c-4891-9818-0713262b0761 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:01.000047 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff708ed-e24c-4891-9818-0713262b0761 None admin] PATH_INFO: `/v3/services` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:01.000698 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ff708ed-e24c-4891-9818-0713262b0761 None admin] RBAC: Adding query filter params (name=image) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:44:01.001025 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ff708ed-e24c-4891-9818-0713262b0761 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:01.002438 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ff708ed-e24c-4891-9818-0713262b0761 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:01.013371 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 524/1028] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:44:00 2022] GET /identity/v3/services?name=image => generated 122 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:44:01.014996 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55532629-0546-4c16-828b-3c8b5b72ee04 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:01.041805 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:01.102287 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-034d61cd-fe39-43b4-bd31-61ab2c0d22c0 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:01.102797 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-034d61cd-fe39-43b4-bd31-61ab2c0d22c0 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:01.102797 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-034d61cd-fe39-43b4-bd31-61ab2c0d22c0 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:01.104331 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 525/1029] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:44:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 09 00:44:01.109233 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01a8e43-e3f7-4b01-a7e3-6f2b2641640b None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:01.109511 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01a8e43-e3f7-4b01-a7e3-6f2b2641640b None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:01.109782 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01a8e43-e3f7-4b01-a7e3-6f2b2641640b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:01.128064 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-b01a8e43-e3f7-4b01-a7e3-6f2b2641640b None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:44:01.129509 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-b01a8e43-e3f7-4b01-a7e3-6f2b2641640b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:44:01.134059 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55532629-0546-4c16-828b-3c8b5b72ee04 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:01.170999 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b55f2709-a6b4-47da-81e3-37db325d48a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:01.171628 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b55f2709-a6b4-47da-81e3-37db325d48a5 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:01.172003 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b55f2709-a6b4-47da-81e3-37db325d48a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:01.172338 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b55f2709-a6b4-47da-81e3-37db325d48a5 None admin] PATH_INFO: `/v3/services` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:01.173030 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b55f2709-a6b4-47da-81e3-37db325d48a5 None admin] RBAC: Adding query filter params (type=image) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:44:01.173329 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b55f2709-a6b4-47da-81e3-37db325d48a5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:01.174682 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b55f2709-a6b4-47da-81e3-37db325d48a5 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:01.183752 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 505/1030] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:44:01 2022] GET /identity/v3/services?type=image => generated 350 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:44:01.186278 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b55f2709-a6b4-47da-81e3-37db325d48a5 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:01.206686 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 526/1031] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:44:01 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 09 00:44:01.275676 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:01.288794 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b55f2709-a6b4-47da-81e3-37db325d48a5 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:01.323099 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f0f0110-942e-4d69-9741-00e714f6bad3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:01.323737 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0f0110-942e-4d69-9741-00e714f6bad3 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:01.324019 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0f0110-942e-4d69-9741-00e714f6bad3 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:01.324293 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0f0110-942e-4d69-9741-00e714f6bad3 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:01.324980 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f0f0110-942e-4d69-9741-00e714f6bad3 None admin] RBAC: Adding query filter params (interface=public, service_id=a743f2b0d4664ed1a2173203d70b09c9, region_id=RegionOne) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:44:01.325286 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f0f0110-942e-4d69-9741-00e714f6bad3 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:01.326637 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f0f0110-942e-4d69-9741-00e714f6bad3 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:01.331736 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 506/1032] 10.222.0.33 () {56 vars in 1251 bytes} [Sun Oct 9 00:44:01 2022] GET /identity/v3/endpoints?service_id=a743f2b0d4664ed1a2173203d70b09c9&interface=public®ion_id=RegionOne => generated 194 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:44:01.337255 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b01a8e43-e3f7-4b01-a7e3-6f2b2641640b None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:01.372010 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:01.437442 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b01a8e43-e3f7-4b01-a7e3-6f2b2641640b None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:01.471392 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63e01e3d-09c9-4b7f-bfdb-3ec38f6b02c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:01.472040 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e01e3d-09c9-4b7f-bfdb-3ec38f6b02c6 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:01.472323 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e01e3d-09c9-4b7f-bfdb-3ec38f6b02c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:01.472599 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e01e3d-09c9-4b7f-bfdb-3ec38f6b02c6 None admin] PATH_INFO: `/v3/services` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:01.473197 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63e01e3d-09c9-4b7f-bfdb-3ec38f6b02c6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:01.474595 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63e01e3d-09c9-4b7f-bfdb-3ec38f6b02c6 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:01.476795 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f0f0110-942e-4d69-9741-00e714f6bad3 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:01.483943 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 527/1033] 10.222.0.33 () {56 vars in 1087 bytes} [Sun Oct 9 00:44:01 2022] GET /identity/v3/services => generated 1957 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:44:01.590672 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f0f0110-942e-4d69-9741-00e714f6bad3 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:01.627196 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44887250-27c0-4330-8f16-4b61cd701571 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:01.627883 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44887250-27c0-4330-8f16-4b61cd701571 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:01.628179 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44887250-27c0-4330-8f16-4b61cd701571 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:01.628434 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44887250-27c0-4330-8f16-4b61cd701571 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:01.640274 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44887250-27c0-4330-8f16-4b61cd701571 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:44:01.645127 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-44887250-27c0-4330-8f16-4b61cd701571 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 09 00:44:01.645127 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:44:01.645127 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:44:01.645127 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:44:01.645127 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:44:01.645127 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:44:01.645127 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:44:01.645127 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:44:01.645127 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:44:01.645127 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:44:01.645127 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:44:01.645127 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:44:01.645127 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:44:01.645127 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:44:01.645127 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:44:01.645127 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:44:01.646121 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:44:01.646121 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:44:01.646121 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:44:01.646121 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:44:01.646121 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:44:01.646121 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:44:01.646121 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:44:01.646121 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:44:01.646121 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:44:01.646121 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:44:01.646121 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:44:01.646121 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:44:01.646121 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:44:01.646121 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:44:01.646121 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:44:01.646121 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:44:01.646996 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:44:01.646996 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:44:01.646996 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:44:01.646996 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:44:01.646996 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:44:01.646996 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:44:01.646996 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:44:01.646996 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:44:01.646996 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:44:01.646996 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:44:01.646996 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 09 00:44:01.646996 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:44:01.646996 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 507/1034] 10.222.0.33 () {56 vars in 1120 bytes} [Sun Oct 9 00:44:01 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 09 00:44:01.652841 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63e01e3d-09c9-4b7f-bfdb-3ec38f6b02c6 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:01.758067 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:01.761500 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63e01e3d-09c9-4b7f-bfdb-3ec38f6b02c6 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:01.797737 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9af61d41-cfb0-421b-8438-f47b4f3fec8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:01.798338 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9af61d41-cfb0-421b-8438-f47b4f3fec8a None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:01.798611 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9af61d41-cfb0-421b-8438-f47b4f3fec8a None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:01.798879 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9af61d41-cfb0-421b-8438-f47b4f3fec8a None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:01.799526 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9af61d41-cfb0-421b-8438-f47b4f3fec8a None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:44:01.799852 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9af61d41-cfb0-421b-8438-f47b4f3fec8a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:01.801198 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9af61d41-cfb0-421b-8438-f47b4f3fec8a None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:01.809584 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 528/1035] 10.222.0.33 () {56 vars in 1115 bytes} [Sun Oct 9 00:44:01 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:44:01.815209 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44887250-27c0-4330-8f16-4b61cd701571 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:01.830233 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:01.915965 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44887250-27c0-4330-8f16-4b61cd701571 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:01.949755 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a910913-c8c8-4b77-a7b7-6716a32bfdde None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:01.950378 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a910913-c8c8-4b77-a7b7-6716a32bfdde None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:01.950653 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a910913-c8c8-4b77-a7b7-6716a32bfdde None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:01.950919 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a910913-c8c8-4b77-a7b7-6716a32bfdde None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:01.961691 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a910913-c8c8-4b77-a7b7-6716a32bfdde None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:01.963122 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a910913-c8c8-4b77-a7b7-6716a32bfdde None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:01.978540 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-1a910913-c8c8-4b77-a7b7-6716a32bfdde None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:44:01.978540 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:44:01.978540 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:44:01.978540 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:44:01.978540 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:44:01.978540 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:44:01.978540 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:44:01.978540 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:44:01.978540 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:44:01.978540 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:44:01.978540 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:44:01.978540 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:44:01.978540 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:44:01.978540 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:44:01.978540 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:44:01.978540 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:44:01.979569 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:44:01.979569 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:44:01.979569 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:44:01.979569 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:44:01.979569 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:44:01.979569 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:44:01.979569 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:44:01.979569 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:44:01.979569 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:44:01.979569 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:44:01.979569 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:44:01.979569 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:44:01.979569 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:44:01.979569 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:44:01.979569 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:44:01.979569 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:44:01.979569 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:44:01.980830 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:44:01.980830 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:44:01.980830 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:44:01.983906 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 508/1036] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:44:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:44:01.985460 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9af61d41-cfb0-421b-8438-f47b4f3fec8a None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:02.095823 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9af61d41-cfb0-421b-8438-f47b4f3fec8a None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:02.130013 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8059e0a-dae9-4e8e-a04c-c09e2a785b3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:02.130641 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8059e0a-dae9-4e8e-a04c-c09e2a785b3d None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:02.130885 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8059e0a-dae9-4e8e-a04c-c09e2a785b3d None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:02.131158 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8059e0a-dae9-4e8e-a04c-c09e2a785b3d None admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:02.132349 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8059e0a-dae9-4e8e-a04c-c09e2a785b3d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:44:02.132648 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8059e0a-dae9-4e8e-a04c-c09e2a785b3d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:02.134001 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8059e0a-dae9-4e8e-a04c-c09e2a785b3d None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:02.146432 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 529/1037] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:44:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:44:02.152615 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a910913-c8c8-4b77-a7b7-6716a32bfdde None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:02.215927 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:02.255337 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a910913-c8c8-4b77-a7b7-6716a32bfdde None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:02.288992 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2576c43e-df02-433c-888a-55e4facd3b62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:02.289865 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2576c43e-df02-433c-888a-55e4facd3b62 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:02.290141 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2576c43e-df02-433c-888a-55e4facd3b62 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:02.290407 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2576c43e-df02-433c-888a-55e4facd3b62 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:02.297809 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2576c43e-df02-433c-888a-55e4facd3b62 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:02.299216 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2576c43e-df02-433c-888a-55e4facd3b62 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:02.310573 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-2576c43e-df02-433c-888a-55e4facd3b62 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 09 00:44:02.310573 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:44:02.310573 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:44:02.310573 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:44:02.310573 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:44:02.310573 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:44:02.310573 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:44:02.310573 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:44:02.310573 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:44:02.310573 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:44:02.310573 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:44:02.310573 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:44:02.310573 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 09 00:44:02.310573 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 09 00:44:02.310573 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 09 00:44:02.310573 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 09 00:44:02.311510 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:44:02.311510 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:44:02.311510 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 09 00:44:02.311510 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 09 00:44:02.311510 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 09 00:44:02.311510 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 09 00:44:02.311510 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:44:02.311510 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:44:02.311510 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:44:02.311510 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:44:02.311510 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:44:02.311510 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:44:02.311510 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:44:02.311510 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:44:02.311510 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:44:02.311510 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:44:02.311510 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:44:02.311510 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:44:02.312770 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:44:02.312770 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:44:02.312770 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 09 00:44:02.312770 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 09 00:44:02.312770 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 09 00:44:02.312770 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 09 00:44:02.312770 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 09 00:44:02.312770 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 09 00:44:02.312770 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 09 00:44:02.312770 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:44:02.316945 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 509/1038] 10.222.0.33 () {56 vars in 1093 bytes} [Sun Oct 9 00:44:02 2022] GET /identity/v3/users/nova => generated 82 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:44:02.317930 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8059e0a-dae9-4e8e-a04c-c09e2a785b3d None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:02.348182 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:02.433322 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8059e0a-dae9-4e8e-a04c-c09e2a785b3d None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:02.462945 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-895a4eac-2d6f-40d8-ab33-511e439c5602 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:02.463233 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-895a4eac-2d6f-40d8-ab33-511e439c5602 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:02.463505 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-895a4eac-2d6f-40d8-ab33-511e439c5602 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:02.464995 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 510/1039] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:44:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 09 00:44:02.469246 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-422d1814-4d31-4bb5-ab3c-f643d220438b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:02.469633 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b026574-f34f-47cc-8561-a078e95c7f39 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:02.469798 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-422d1814-4d31-4bb5-ab3c-f643d220438b None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:02.469973 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b026574-f34f-47cc-8561-a078e95c7f39 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:02.470079 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-422d1814-4d31-4bb5-ab3c-f643d220438b None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:02.470235 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b026574-f34f-47cc-8561-a078e95c7f39 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:02.470327 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-422d1814-4d31-4bb5-ab3c-f643d220438b None admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:02.471028 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-422d1814-4d31-4bb5-ab3c-f643d220438b None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:44:02.471332 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-422d1814-4d31-4bb5-ab3c-f643d220438b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:02.472837 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-422d1814-4d31-4bb5-ab3c-f643d220438b None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:02.491817 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-8b026574-f34f-47cc-8561-a078e95c7f39 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:44:02.499465 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 530/1040] 10.222.0.33 () {56 vars in 1133 bytes} [Sun Oct 9 00:44:02 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:44:02.499671 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-8b026574-f34f-47cc-8561-a078e95c7f39 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:44:02.501599 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-422d1814-4d31-4bb5-ab3c-f643d220438b None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:02.568381 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 511/1041] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:44:02 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 09 00:44:02.597718 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:02.618478 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-422d1814-4d31-4bb5-ab3c-f643d220438b None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:02.652783 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c7c5870-4d60-4ee8-a87e-6b411fe433fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:02.653379 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7c5870-4d60-4ee8-a87e-6b411fe433fc None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:02.653654 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7c5870-4d60-4ee8-a87e-6b411fe433fc None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:02.653921 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7c5870-4d60-4ee8-a87e-6b411fe433fc None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:02.723163 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c7c5870-4d60-4ee8-a87e-6b411fe433fc None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:02.724541 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c7c5870-4d60-4ee8-a87e-6b411fe433fc None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:02.804487 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:02.826443 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 531/1042] 10.222.0.33 () {58 vars in 1282 bytes} [Sun Oct 9 00:44:02 2022] GET /identity/v3/auth/tokens => generated 1969 bytes in 326 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Oct 09 00:44:02.832163 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b026574-f34f-47cc-8561-a078e95c7f39 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:02.836457 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c7c5870-4d60-4ee8-a87e-6b411fe433fc None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:02.892554 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:02.938496 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b026574-f34f-47cc-8561-a078e95c7f39 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:02.942040 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c7c5870-4d60-4ee8-a87e-6b411fe433fc None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:02.972023 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5a2b9fb-21ce-4db0-86e3-a34c56441d11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:02.972656 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a2b9fb-21ce-4db0-86e3-a34c56441d11 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:02.972936 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a2b9fb-21ce-4db0-86e3-a34c56441d11 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:02.973282 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a2b9fb-21ce-4db0-86e3-a34c56441d11 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:02.979734 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c50d13bf-6bfe-49bf-a3cc-90bdcc4efaa6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:02.980348 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c50d13bf-6bfe-49bf-a3cc-90bdcc4efaa6 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:02.980624 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c50d13bf-6bfe-49bf-a3cc-90bdcc4efaa6 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:02.981271 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c50d13bf-6bfe-49bf-a3cc-90bdcc4efaa6 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:02.984646 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5a2b9fb-21ce-4db0-86e3-a34c56441d11 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:02.986068 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5a2b9fb-21ce-4db0-86e3-a34c56441d11 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:02.992319 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c50d13bf-6bfe-49bf-a3cc-90bdcc4efaa6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:44:02.996992 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-c50d13bf-6bfe-49bf-a3cc-90bdcc4efaa6 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 09 00:44:02.996992 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:44:02.996992 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:44:02.996992 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:44:02.996992 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:44:02.996992 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:44:02.996992 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:44:02.996992 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:44:02.996992 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:44:02.996992 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:44:02.996992 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:44:02.996992 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:44:02.996992 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:44:02.996992 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:44:02.996992 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:44:02.996992 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:44:02.997910 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:44:02.997910 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:44:02.997910 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:44:02.997910 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:44:02.997910 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:44:02.997910 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:44:02.997910 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:44:02.997910 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:44:02.997910 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:44:02.997910 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:44:02.997910 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:44:02.997910 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:44:02.997910 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:44:02.997910 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:44:02.997910 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:44:02.997910 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:44:02.998767 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:44:02.998767 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:44:02.998767 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:44:02.998767 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:44:02.998767 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:44:02.998767 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:44:02.998767 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:44:02.998767 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:44:02.998767 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:44:02.998767 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:44:02.998767 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 09 00:44:02.998767 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:44:03.001917 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-f5a2b9fb-21ce-4db0-86e3-a34c56441d11 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:44:03.001917 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:44:03.001917 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:44:03.001917 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:44:03.001917 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:44:03.001917 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:44:03.001917 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:44:03.001917 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:44:03.001917 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:44:03.001917 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:44:03.001917 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:44:03.001917 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:44:03.001917 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:44:03.001917 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:44:03.001917 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:44:03.001917 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:44:03.008293 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:44:03.008293 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:44:03.008293 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:44:03.008293 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:44:03.008293 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:44:03.008293 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:44:03.008293 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:44:03.008293 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:44:03.008293 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:44:03.008293 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:44:03.008293 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:44:03.008293 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:44:03.008293 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:44:03.008293 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:44:03.008293 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:44:03.008293 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:44:03.008293 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:44:03.009199 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:44:03.009199 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:44:03.009199 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:44:03.009382 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 532/1043] 10.222.0.33 () {56 vars in 1096 bytes} [Sun Oct 9 00:44:02 2022] GET /identity/v3/roles/admin => generated 83 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 09 00:44:03.009455 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 512/1044] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:44:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 09 00:44:03.009524 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c50d13bf-6bfe-49bf-a3cc-90bdcc4efaa6 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:03.009593 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5a2b9fb-21ce-4db0-86e3-a34c56441d11 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:03.048312 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:03.112971 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c50d13bf-6bfe-49bf-a3cc-90bdcc4efaa6 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:03.123761 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5a2b9fb-21ce-4db0-86e3-a34c56441d11 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:03.130671 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:03.151196 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-527f1d8e-1019-404b-96cb-34b41543fbcb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:03.151916 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527f1d8e-1019-404b-96cb-34b41543fbcb None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:03.152202 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527f1d8e-1019-404b-96cb-34b41543fbcb None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:03.152476 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527f1d8e-1019-404b-96cb-34b41543fbcb None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:03.153144 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-527f1d8e-1019-404b-96cb-34b41543fbcb None admin] RBAC: Adding query filter params (name=admin) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:44:03.153445 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-527f1d8e-1019-404b-96cb-34b41543fbcb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:03.154819 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-527f1d8e-1019-404b-96cb-34b41543fbcb None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:03.161232 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aee954b-aafd-4ff1-8379-4bde7442c6b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:03.162109 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aee954b-aafd-4ff1-8379-4bde7442c6b6 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:03.162386 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aee954b-aafd-4ff1-8379-4bde7442c6b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:03.162658 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aee954b-aafd-4ff1-8379-4bde7442c6b6 None admin] PATH_INFO: `/v3/domains` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:03.163330 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aee954b-aafd-4ff1-8379-4bde7442c6b6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:44:03.163662 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aee954b-aafd-4ff1-8379-4bde7442c6b6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:03.164839 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 533/1045] 10.222.0.33 () {56 vars in 1099 bytes} [Sun Oct 9 00:44:03 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:44:03.165277 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aee954b-aafd-4ff1-8379-4bde7442c6b6 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:03.170686 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-527f1d8e-1019-404b-96cb-34b41543fbcb None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:03.179255 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 513/1046] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:44:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:44:03.184347 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aee954b-aafd-4ff1-8379-4bde7442c6b6 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:03.276923 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-527f1d8e-1019-404b-96cb-34b41543fbcb None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:03.278357 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:03.290426 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aee954b-aafd-4ff1-8379-4bde7442c6b6 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:03.315340 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5485b6d1-4361-405f-a25b-3204d04d3f00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:03.315987 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5485b6d1-4361-405f-a25b-3204d04d3f00 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:03.316265 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5485b6d1-4361-405f-a25b-3204d04d3f00 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:03.316531 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5485b6d1-4361-405f-a25b-3204d04d3f00 None admin] PATH_INFO: `/v3/users/c2fd402ae6f74c89a66e314d5013e78e` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:03.325939 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e41580fd-5e6d-4533-8cec-92eba78bdeb4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:03.326566 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e41580fd-5e6d-4533-8cec-92eba78bdeb4 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:03.326847 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e41580fd-5e6d-4533-8cec-92eba78bdeb4 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:03.327118 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e41580fd-5e6d-4533-8cec-92eba78bdeb4 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:03.330469 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5485b6d1-4361-405f-a25b-3204d04d3f00 None admin] RBAC: Authorizing `identity:get_user(user_id=c2fd402ae6f74c89a66e314d5013e78e)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:03.332011 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5485b6d1-4361-405f-a25b-3204d04d3f00 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:03.333781 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 534/1047] 10.222.0.33 () {56 vars in 1177 bytes} [Sun Oct 9 00:44:03 2022] GET /identity/v3/users/c2fd402ae6f74c89a66e314d5013e78e => generated 285 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:44:03.335882 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e41580fd-5e6d-4533-8cec-92eba78bdeb4 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:03.337310 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e41580fd-5e6d-4533-8cec-92eba78bdeb4 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:03.339644 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5485b6d1-4361-405f-a25b-3204d04d3f00 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:03.349678 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-e41580fd-5e6d-4533-8cec-92eba78bdeb4 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 09 00:44:03.349678 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:44:03.349678 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:44:03.349678 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:44:03.349678 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:44:03.349678 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:44:03.349678 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:44:03.349678 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:44:03.349678 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:44:03.349678 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:44:03.349678 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:44:03.349678 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:44:03.349678 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 09 00:44:03.349678 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 09 00:44:03.349678 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 09 00:44:03.350844 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 09 00:44:03.350844 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:44:03.350844 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:44:03.350844 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:44:03.350844 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:44:03.350844 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:44:03.350844 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:44:03.350844 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:44:03.350844 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:44:03.350844 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:44:03.350844 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:44:03.350844 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:44:03.350844 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:44:03.350844 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:44:03.350844 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:44:03.350844 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:44:03.350844 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:44:03.351820 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 09 00:44:03.351820 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 09 00:44:03.351820 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 09 00:44:03.351820 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 09 00:44:03.351820 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 09 00:44:03.351820 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 09 00:44:03.351820 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 09 00:44:03.351820 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:44:03.351820 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 514/1048] 10.222.0.33 () {56 vars in 1111 bytes} [Sun Oct 9 00:44:03 2022] GET /identity/v3/projects/service => generated 88 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:44:03.356620 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e41580fd-5e6d-4533-8cec-92eba78bdeb4 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:03.387330 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:03.454435 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5485b6d1-4361-405f-a25b-3204d04d3f00 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:03.463049 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e41580fd-5e6d-4533-8cec-92eba78bdeb4 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:03.489590 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9fe2a0c-0963-41b2-80b1-cca2fe277739 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:03.490193 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9fe2a0c-0963-41b2-80b1-cca2fe277739 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:03.490477 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9fe2a0c-0963-41b2-80b1-cca2fe277739 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:03.490870 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9fe2a0c-0963-41b2-80b1-cca2fe277739 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:03.499871 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4e6b1bf-d04f-4c85-a0ae-cd513ad16b05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:03.500474 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4e6b1bf-d04f-4c85-a0ae-cd513ad16b05 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:03.500753 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4e6b1bf-d04f-4c85-a0ae-cd513ad16b05 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:03.501022 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4e6b1bf-d04f-4c85-a0ae-cd513ad16b05 None admin] PATH_INFO: `/v3/projects` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:03.501711 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4e6b1bf-d04f-4c85-a0ae-cd513ad16b05 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:44:03.502010 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4e6b1bf-d04f-4c85-a0ae-cd513ad16b05 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:03.503421 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4e6b1bf-d04f-4c85-a0ae-cd513ad16b05 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:03.518331 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 515/1049] 10.222.0.33 () {56 vars in 1148 bytes} [Sun Oct 9 00:44:03 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:44:03.520962 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:03.525291 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4e6b1bf-d04f-4c85-a0ae-cd513ad16b05 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:03.567617 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9fe2a0c-0963-41b2-80b1-cca2fe277739 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:03.569014 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9fe2a0c-0963-41b2-80b1-cca2fe277739 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:03.627207 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:03.644128 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4e6b1bf-d04f-4c85-a0ae-cd513ad16b05 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:03.671242 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 535/1050] 10.222.0.33 () {58 vars in 1282 bytes} [Sun Oct 9 00:44:03 2022] GET /identity/v3/auth/tokens => generated 1969 bytes in 332 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 09 00:44:03.672524 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e4dfb2b-572e-468a-bec9-61c9a209117d None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:03.672810 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e4dfb2b-572e-468a-bec9-61c9a209117d None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:03.673077 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e4dfb2b-572e-468a-bec9-61c9a209117d None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:03.674282 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 536/1051] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:44:03 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:44:03.677056 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e4dfb2b-572e-468a-bec9-61c9a209117d None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:03.684310 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e328b39-6713-4035-b585-03f2f5d02f36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:03.684915 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e328b39-6713-4035-b585-03f2f5d02f36 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:03.685195 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e328b39-6713-4035-b585-03f2f5d02f36 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:03.685834 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e328b39-6713-4035-b585-03f2f5d02f36 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:03.686179 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e328b39-6713-4035-b585-03f2f5d02f36 None admin] RBAC: Adding query filter params (role.id=fddb7718710e48bb8034bd9390b820a2, scope.project.id=013c83476b1340fbb051061833c4a9c6, user.id=e0a5d95969e74157bb2d2a57207e1651) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:44:03.686486 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e328b39-6713-4035-b585-03f2f5d02f36 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:03.687932 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e328b39-6713-4035-b585-03f2f5d02f36 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:03.693091 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 516/1052] 10.222.0.33 () {56 vars in 1374 bytes} [Sun Oct 9 00:44:03 2022] GET /identity/v3/role_assignments?user.id=e0a5d95969e74157bb2d2a57207e1651&scope.project.id=013c83476b1340fbb051061833c4a9c6&role.id=fddb7718710e48bb8034bd9390b820a2 => generated 259 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:44:03.694231 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2222b9e3-1d3f-45d4-9584-c3e56556c11e None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:03.694516 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2222b9e3-1d3f-45d4-9584-c3e56556c11e None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:03.694801 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2222b9e3-1d3f-45d4-9584-c3e56556c11e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:03.713277 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-2222b9e3-1d3f-45d4-9584-c3e56556c11e None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:44:03.714501 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-2222b9e3-1d3f-45d4-9584-c3e56556c11e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:44:03.744785 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:03.792239 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 517/1053] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:44:03 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 09 00:44:03.794163 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e4dfb2b-572e-468a-bec9-61c9a209117d None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:03.830309 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-404bb8cf-b3be-4080-bd55-70b5bd479a40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:03.830854 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-404bb8cf-b3be-4080-bd55-70b5bd479a40 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:03.831130 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-404bb8cf-b3be-4080-bd55-70b5bd479a40 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:03.831398 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-404bb8cf-b3be-4080-bd55-70b5bd479a40 None admin] PATH_INFO: `/v3/projects/0781f36cb6fd4f61b57eb625b8420b87` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:03.832537 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-404bb8cf-b3be-4080-bd55-70b5bd479a40 None admin] RBAC: Authorizing `identity:get_project(project_id=0781f36cb6fd4f61b57eb625b8420b87)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:03.833991 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-404bb8cf-b3be-4080-bd55-70b5bd479a40 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:03.836493 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 537/1054] 10.222.0.33 () {56 vars in 1186 bytes} [Sun Oct 9 00:44:03 2022] GET /identity/v3/projects/0781f36cb6fd4f61b57eb625b8420b87 => generated 311 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:44:03.842410 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2222b9e3-1d3f-45d4-9584-c3e56556c11e None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:03.926202 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:03.946111 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2222b9e3-1d3f-45d4-9584-c3e56556c11e None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:03.981577 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-effedef6-6afd-4f28-8245-7c3c8227ea6b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:03.982184 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-effedef6-6afd-4f28-8245-7c3c8227ea6b None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:03.982465 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-effedef6-6afd-4f28-8245-7c3c8227ea6b None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:03.982739 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-effedef6-6afd-4f28-8245-7c3c8227ea6b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:03.983452 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-effedef6-6afd-4f28-8245-7c3c8227ea6b None admin] RBAC: Adding query filter params (role.id=092036ee46ac427baf4d56468575f697, scope.project.id=0781f36cb6fd4f61b57eb625b8420b87, user.id=c2fd402ae6f74c89a66e314d5013e78e) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:44:03.983796 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-effedef6-6afd-4f28-8245-7c3c8227ea6b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:03.985207 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-effedef6-6afd-4f28-8245-7c3c8227ea6b None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:03.995892 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 518/1055] 10.222.0.33 () {56 vars in 1374 bytes} [Sun Oct 9 00:44:03 2022] GET /identity/v3/role_assignments?user.id=c2fd402ae6f74c89a66e314d5013e78e&scope.project.id=0781f36cb6fd4f61b57eb625b8420b87&role.id=092036ee46ac427baf4d56468575f697 => generated 259 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:44:04.061792 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-404bb8cf-b3be-4080-bd55-70b5bd479a40 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:04.179141 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-404bb8cf-b3be-4080-bd55-70b5bd479a40 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:04.207048 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:04.216046 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c62d450b-7e01-4ebd-82cb-1b116be3db02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:04.216680 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c62d450b-7e01-4ebd-82cb-1b116be3db02 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:04.216984 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c62d450b-7e01-4ebd-82cb-1b116be3db02 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:04.217232 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c62d450b-7e01-4ebd-82cb-1b116be3db02 None admin] PATH_INFO: `/v3/services/image` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:04.217809 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c62d450b-7e01-4ebd-82cb-1b116be3db02 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:04.219168 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c62d450b-7e01-4ebd-82cb-1b116be3db02 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:04.227866 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-c62d450b-7e01-4ebd-82cb-1b116be3db02 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Oct 09 00:44:04.227866 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:44:04.227866 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 09 00:44:04.227866 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 09 00:44:04.227866 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 09 00:44:04.227866 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 09 00:44:04.227866 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 09 00:44:04.227866 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 09 00:44:04.227866 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 09 00:44:04.227866 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:44:04.227866 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 09 00:44:04.227866 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:44:04.227866 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:44:04.227866 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:44:04.227866 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:44:04.227866 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:44:04.227866 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:44:04.228915 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:44:04.228915 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:44:04.228915 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:44:04.228915 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:44:04.228915 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:44:04.228915 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:44:04.228915 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 09 00:44:04.228915 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 09 00:44:04.228915 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 09 00:44:04.228915 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 09 00:44:04.228915 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:44:04.228915 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:44:04.228915 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:44:04.228915 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:44:04.228915 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:44:04.228915 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:44:04.229747 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:44:04.229747 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:44:04.229747 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:44:04.229747 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:44:04.229747 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:44:04.229747 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:44:04.229747 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:44:04.229747 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:44:04.229747 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:44:04.229747 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:44:04.229747 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 09 00:44:04.229747 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 09 00:44:04.229747 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 09 00:44:04.229747 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:44:04.231666 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 538/1056] 10.222.0.33 () {56 vars in 1105 bytes} [Sun Oct 9 00:44:04 2022] GET /identity/v3/services/image => generated 86 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:44:04.234897 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-effedef6-6afd-4f28-8245-7c3c8227ea6b None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:04.335992 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-effedef6-6afd-4f28-8245-7c3c8227ea6b None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:04.368831 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba9381ed-e29d-4cba-ae1c-428acf8df295 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:04.369431 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9381ed-e29d-4cba-ae1c-428acf8df295 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:04.369707 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9381ed-e29d-4cba-ae1c-428acf8df295 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:04.369974 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9381ed-e29d-4cba-ae1c-428acf8df295 None admin] PATH_INFO: `/v3/services` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:04.370585 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba9381ed-e29d-4cba-ae1c-428acf8df295 None admin] RBAC: Adding query filter params (name=image) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:44:04.370882 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba9381ed-e29d-4cba-ae1c-428acf8df295 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:04.372255 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba9381ed-e29d-4cba-ae1c-428acf8df295 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:04.377114 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 519/1057] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:44:04 2022] GET /identity/v3/services?name=image => generated 122 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:44:04.383077 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c62d450b-7e01-4ebd-82cb-1b116be3db02 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:04.483688 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c62d450b-7e01-4ebd-82cb-1b116be3db02 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:04.516901 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64dd6bfe-0b8f-4d6a-963d-46adc20048f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:04.517497 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64dd6bfe-0b8f-4d6a-963d-46adc20048f9 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:04.517772 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64dd6bfe-0b8f-4d6a-963d-46adc20048f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:04.518038 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64dd6bfe-0b8f-4d6a-963d-46adc20048f9 None admin] PATH_INFO: `/v3/services` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:04.518651 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64dd6bfe-0b8f-4d6a-963d-46adc20048f9 None admin] RBAC: Adding query filter params (type=image) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:44:04.518945 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64dd6bfe-0b8f-4d6a-963d-46adc20048f9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:04.520326 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64dd6bfe-0b8f-4d6a-963d-46adc20048f9 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:04.530018 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 539/1058] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:44:04 2022] GET /identity/v3/services?type=image => generated 350 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:44:04.531240 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba9381ed-e29d-4cba-ae1c-428acf8df295 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:04.543245 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:04.633533 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba9381ed-e29d-4cba-ae1c-428acf8df295 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:04.667379 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9c3fde5-0f5c-42a5-8fcb-81850e70a667 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:04.668003 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c3fde5-0f5c-42a5-8fcb-81850e70a667 None admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:04.668279 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c3fde5-0f5c-42a5-8fcb-81850e70a667 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:04.668547 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c3fde5-0f5c-42a5-8fcb-81850e70a667 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:04.669206 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9c3fde5-0f5c-42a5-8fcb-81850e70a667 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:04.670547 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9c3fde5-0f5c-42a5-8fcb-81850e70a667 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:04.691512 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 520/1059] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:44:04 2022] POST /identity/v3/endpoints => generated 327 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:44:05.833449 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c36c67-9afd-4627-84f9-c12b6ee2fa55 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:05.834019 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c36c67-9afd-4627-84f9-c12b6ee2fa55 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:05.834019 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c36c67-9afd-4627-84f9-c12b6ee2fa55 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:05.835516 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 540/1060] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:44:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:44:05.840340 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c3050be-48fe-4283-98ed-798f080e4a69 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:05.840719 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c3050be-48fe-4283-98ed-798f080e4a69 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:05.840895 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c3050be-48fe-4283-98ed-798f080e4a69 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:05.859277 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-0c3050be-48fe-4283-98ed-798f080e4a69 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:44:05.860464 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-0c3050be-48fe-4283-98ed-798f080e4a69 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:44:05.919945 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:05.937077 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 521/1061] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:44:05 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 09 00:44:06.165817 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be90d51-f04b-403f-8042-6f8e7b793b47 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:06.166201 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be90d51-f04b-403f-8042-6f8e7b793b47 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:06.166359 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be90d51-f04b-403f-8042-6f8e7b793b47 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:06.167850 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 541/1062] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:44:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:44:06.173053 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be12d422-ff1c-499c-9343-13a7d8de0c3b None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:06.173342 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be12d422-ff1c-499c-9343-13a7d8de0c3b None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:06.173614 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be12d422-ff1c-499c-9343-13a7d8de0c3b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:06.191891 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-be12d422-ff1c-499c-9343-13a7d8de0c3b None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:44:06.193065 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-be12d422-ff1c-499c-9343-13a7d8de0c3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:44:06.202609 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2be90d51-f04b-403f-8042-6f8e7b793b47 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:06.263219 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 522/1063] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:44:06 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 09 00:44:06.267961 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:06.315534 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2be90d51-f04b-403f-8042-6f8e7b793b47 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:06.348939 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c852073-541b-4621-9e8d-d05de57cc012 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:06.349561 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c852073-541b-4621-9e8d-d05de57cc012 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:06.349838 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c852073-541b-4621-9e8d-d05de57cc012 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:06.350105 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c852073-541b-4621-9e8d-d05de57cc012 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:06.360600 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c852073-541b-4621-9e8d-d05de57cc012 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:44:06.365315 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-8c852073-541b-4621-9e8d-d05de57cc012 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 09 00:44:06.365315 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:44:06.365315 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:44:06.365315 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:44:06.365315 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:44:06.365315 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:44:06.365315 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:44:06.365315 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:44:06.365315 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:44:06.365315 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:44:06.365315 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:44:06.365315 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:44:06.365315 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:44:06.365315 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:44:06.365315 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:44:06.365315 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:44:06.366736 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:44:06.366736 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:44:06.366736 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:44:06.366736 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:44:06.366736 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:44:06.366736 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:44:06.366736 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:44:06.366736 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:44:06.366736 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:44:06.366736 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:44:06.366736 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:44:06.366736 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:44:06.366736 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:44:06.366736 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:44:06.366736 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:44:06.366736 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:44:06.367570 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:44:06.367570 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:44:06.367570 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:44:06.367570 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:44:06.367570 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:44:06.367570 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:44:06.367570 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:44:06.367570 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:44:06.367570 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:44:06.367570 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:44:06.367570 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 09 00:44:06.367570 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:44:06.367570 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 542/1064] 10.222.0.33 () {56 vars in 1120 bytes} [Sun Oct 9 00:44:06 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 09 00:44:06.376822 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be12d422-ff1c-499c-9343-13a7d8de0c3b None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:06.476345 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be12d422-ff1c-499c-9343-13a7d8de0c3b None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:06.496742 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:06.511783 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70c34849-41c8-4749-9dd8-b19a89b647e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:06.512392 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70c34849-41c8-4749-9dd8-b19a89b647e2 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:06.512671 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70c34849-41c8-4749-9dd8-b19a89b647e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:06.512942 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70c34849-41c8-4749-9dd8-b19a89b647e2 None admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:06.513592 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70c34849-41c8-4749-9dd8-b19a89b647e2 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:44:06.513893 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70c34849-41c8-4749-9dd8-b19a89b647e2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:06.515250 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70c34849-41c8-4749-9dd8-b19a89b647e2 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:06.527577 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 523/1065] 10.222.0.33 () {56 vars in 1115 bytes} [Sun Oct 9 00:44:06 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:44:06.529082 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c852073-541b-4621-9e8d-d05de57cc012 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:06.538627 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70c34849-41c8-4749-9dd8-b19a89b647e2 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:06.641459 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c852073-541b-4621-9e8d-d05de57cc012 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:06.645379 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70c34849-41c8-4749-9dd8-b19a89b647e2 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:06.663927 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:06.681435 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ebfc0aa-9302-444e-96c6-cce6b1e3e75a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:06.682070 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ebfc0aa-9302-444e-96c6-cce6b1e3e75a None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:06.682352 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ebfc0aa-9302-444e-96c6-cce6b1e3e75a None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:06.682626 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ebfc0aa-9302-444e-96c6-cce6b1e3e75a None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:06.685057 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a13e7c3a-041e-454b-841f-1904ebf1d62b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:06.685684 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13e7c3a-041e-454b-841f-1904ebf1d62b None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:06.685963 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13e7c3a-041e-454b-841f-1904ebf1d62b None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:06.686232 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13e7c3a-041e-454b-841f-1904ebf1d62b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:06.693699 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ebfc0aa-9302-444e-96c6-cce6b1e3e75a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:44:06.697597 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a13e7c3a-041e-454b-841f-1904ebf1d62b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:06.699037 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a13e7c3a-041e-454b-841f-1904ebf1d62b None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:06.699137 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-3ebfc0aa-9302-444e-96c6-cce6b1e3e75a None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 09 00:44:06.699137 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:44:06.699137 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:44:06.699137 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:44:06.699137 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:44:06.699137 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:44:06.699137 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:44:06.699137 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:44:06.699137 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:44:06.699137 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:44:06.699137 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:44:06.699137 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:44:06.699137 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:44:06.699137 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:44:06.699137 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:44:06.699137 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:44:06.700339 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:44:06.700339 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:44:06.700339 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:44:06.700339 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:44:06.700339 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:44:06.700339 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:44:06.700339 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:44:06.700339 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:44:06.700339 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:44:06.700339 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:44:06.700339 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:44:06.700339 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:44:06.700339 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:44:06.700339 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:44:06.700339 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:44:06.700339 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:44:06.701144 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:44:06.701144 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:44:06.701144 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:44:06.701144 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:44:06.701144 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:44:06.701144 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:44:06.701144 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:44:06.701144 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:44:06.701144 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:44:06.701144 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:44:06.701144 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 09 00:44:06.701144 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:44:06.705232 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 524/1066] 10.222.0.33 () {56 vars in 1096 bytes} [Sun Oct 9 00:44:06 2022] GET /identity/v3/roles/admin => generated 83 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 09 00:44:06.706015 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ebfc0aa-9302-444e-96c6-cce6b1e3e75a None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:06.715808 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-a13e7c3a-041e-454b-841f-1904ebf1d62b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:44:06.715808 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:44:06.715808 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:44:06.715808 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:44:06.715808 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:44:06.715808 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:44:06.715808 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:44:06.715808 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:44:06.715808 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:44:06.715808 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:44:06.715808 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:44:06.715808 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:44:06.715808 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:44:06.715808 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:44:06.715808 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:44:06.715808 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:44:06.716752 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:44:06.716752 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:44:06.716752 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:44:06.716752 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:44:06.716752 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:44:06.716752 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:44:06.716752 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:44:06.716752 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:44:06.716752 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:44:06.716752 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:44:06.716752 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:44:06.716752 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:44:06.716752 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:44:06.716752 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:44:06.716752 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:44:06.716752 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:44:06.716752 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:44:06.717708 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:44:06.717708 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:44:06.717708 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:44:06.717708 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 543/1067] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:44:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 09 00:44:06.724280 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a13e7c3a-041e-454b-841f-1904ebf1d62b None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:06.753844 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:06.817635 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ebfc0aa-9302-444e-96c6-cce6b1e3e75a None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:06.839616 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a13e7c3a-041e-454b-841f-1904ebf1d62b None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:06.854712 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a71d84b-8960-4c73-a7be-31e90a846ff4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:06.855316 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a71d84b-8960-4c73-a7be-31e90a846ff4 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:06.855610 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a71d84b-8960-4c73-a7be-31e90a846ff4 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:06.855884 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a71d84b-8960-4c73-a7be-31e90a846ff4 None admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:06.856531 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a71d84b-8960-4c73-a7be-31e90a846ff4 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:44:06.856831 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a71d84b-8960-4c73-a7be-31e90a846ff4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:06.858188 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a71d84b-8960-4c73-a7be-31e90a846ff4 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:06.868008 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 525/1068] 10.222.0.33 () {56 vars in 1099 bytes} [Sun Oct 9 00:44:06 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:44:06.869195 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04165b91-83a0-4871-94bc-41b713092286 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:06.869472 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04165b91-83a0-4871-94bc-41b713092286 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:06.869744 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04165b91-83a0-4871-94bc-41b713092286 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:06.870904 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 526/1069] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:44:06 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:44:06.873758 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04165b91-83a0-4871-94bc-41b713092286 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:06.878788 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6fd903e-8141-424a-8dc7-05a69854e057 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:06.879392 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6fd903e-8141-424a-8dc7-05a69854e057 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:06.879711 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6fd903e-8141-424a-8dc7-05a69854e057 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:06.879989 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6fd903e-8141-424a-8dc7-05a69854e057 None admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:06.880630 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6fd903e-8141-424a-8dc7-05a69854e057 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:44:06.880933 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6fd903e-8141-424a-8dc7-05a69854e057 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:06.882338 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6fd903e-8141-424a-8dc7-05a69854e057 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:06.893857 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 544/1070] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:44:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:44:06.895003 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-354ee55b-3a1f-4fb9-90f1-0639361f306e None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:06.895293 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-354ee55b-3a1f-4fb9-90f1-0639361f306e None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:06.895567 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-354ee55b-3a1f-4fb9-90f1-0639361f306e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:06.915104 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-354ee55b-3a1f-4fb9-90f1-0639361f306e None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:44:06.916325 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-354ee55b-3a1f-4fb9-90f1-0639361f306e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:44:06.938426 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:06.972944 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04165b91-83a0-4871-94bc-41b713092286 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:06.975155 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:06.983088 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 545/1071] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:44:06 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 09 00:44:06.983927 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-354ee55b-3a1f-4fb9-90f1-0639361f306e None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:07.015070 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd71cd18-3ba0-48be-a369-9f5f899468d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:07.015635 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd71cd18-3ba0-48be-a369-9f5f899468d0 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:07.015916 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd71cd18-3ba0-48be-a369-9f5f899468d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:07.016178 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd71cd18-3ba0-48be-a369-9f5f899468d0 None admin] PATH_INFO: `/v3/users/c2fd402ae6f74c89a66e314d5013e78e` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:07.017289 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd71cd18-3ba0-48be-a369-9f5f899468d0 None admin] RBAC: Authorizing `identity:get_user(user_id=c2fd402ae6f74c89a66e314d5013e78e)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:07.018707 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd71cd18-3ba0-48be-a369-9f5f899468d0 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:07.020311 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 527/1072] 10.222.0.33 () {56 vars in 1177 bytes} [Sun Oct 9 00:44:06 2022] GET /identity/v3/users/c2fd402ae6f74c89a66e314d5013e78e => generated 285 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:44:07.026127 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd71cd18-3ba0-48be-a369-9f5f899468d0 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:07.089479 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-354ee55b-3a1f-4fb9-90f1-0639361f306e None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:07.124113 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d45b4fd2-2665-4086-b8db-1e90c95817be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:07.124820 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d45b4fd2-2665-4086-b8db-1e90c95817be None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:07.125095 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d45b4fd2-2665-4086-b8db-1e90c95817be None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:07.125363 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d45b4fd2-2665-4086-b8db-1e90c95817be None admin] PATH_INFO: `/v3/users/nova` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:07.133033 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d45b4fd2-2665-4086-b8db-1e90c95817be None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:07.134473 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d45b4fd2-2665-4086-b8db-1e90c95817be None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:07.145124 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd71cd18-3ba0-48be-a369-9f5f899468d0 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:07.146066 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-d45b4fd2-2665-4086-b8db-1e90c95817be None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 09 00:44:07.146066 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:44:07.146066 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:44:07.146066 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:44:07.146066 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:44:07.146066 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:44:07.146066 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:44:07.146066 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:44:07.146066 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:44:07.146066 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:44:07.146066 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:44:07.146066 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:44:07.146066 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 09 00:44:07.146066 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 09 00:44:07.146066 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 09 00:44:07.146066 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 09 00:44:07.146956 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:44:07.146956 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:44:07.146956 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 09 00:44:07.146956 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 09 00:44:07.146956 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 09 00:44:07.146956 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 09 00:44:07.146956 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:44:07.146956 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:44:07.146956 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:44:07.146956 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:44:07.146956 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:44:07.146956 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:44:07.146956 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:44:07.146956 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:44:07.146956 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:44:07.146956 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:44:07.146956 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:44:07.146956 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:44:07.148001 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:44:07.148001 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:44:07.148001 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 09 00:44:07.148001 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 09 00:44:07.148001 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 09 00:44:07.148001 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 09 00:44:07.148001 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 09 00:44:07.148001 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 09 00:44:07.148001 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 09 00:44:07.148001 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:44:07.148001 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 546/1073] 10.222.0.33 () {56 vars in 1093 bytes} [Sun Oct 9 00:44:06 2022] GET /identity/v3/users/nova => generated 82 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:44:07.152790 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d45b4fd2-2665-4086-b8db-1e90c95817be None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:07.172938 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:07.183862 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ea11ce6-49e1-43b3-a70c-497a07050822 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:07.184472 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ea11ce6-49e1-43b3-a70c-497a07050822 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:07.184747 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ea11ce6-49e1-43b3-a70c-497a07050822 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:07.185026 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ea11ce6-49e1-43b3-a70c-497a07050822 None admin] PATH_INFO: `/v3/projects/0781f36cb6fd4f61b57eb625b8420b87` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:07.186095 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ea11ce6-49e1-43b3-a70c-497a07050822 None admin] RBAC: Authorizing `identity:get_project(project_id=0781f36cb6fd4f61b57eb625b8420b87)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:07.187505 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ea11ce6-49e1-43b3-a70c-497a07050822 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:07.189567 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 528/1074] 10.222.0.33 () {56 vars in 1186 bytes} [Sun Oct 9 00:44:07 2022] GET /identity/v3/projects/0781f36cb6fd4f61b57eb625b8420b87 => generated 311 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:44:07.195308 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ea11ce6-49e1-43b3-a70c-497a07050822 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:07.239645 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:07.255870 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d45b4fd2-2665-4086-b8db-1e90c95817be None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:07.289035 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4a40137-c7f1-451a-8ece-baadda59505a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:07.289812 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4a40137-c7f1-451a-8ece-baadda59505a None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:07.290086 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4a40137-c7f1-451a-8ece-baadda59505a None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:07.290355 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4a40137-c7f1-451a-8ece-baadda59505a None admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:07.291055 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4a40137-c7f1-451a-8ece-baadda59505a None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:44:07.291354 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4a40137-c7f1-451a-8ece-baadda59505a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:07.292807 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4a40137-c7f1-451a-8ece-baadda59505a None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:07.298832 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ea11ce6-49e1-43b3-a70c-497a07050822 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:07.310054 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 547/1075] 10.222.0.33 () {56 vars in 1133 bytes} [Sun Oct 9 00:44:07 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:44:07.310054 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4a40137-c7f1-451a-8ece-baadda59505a None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:07.332536 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d38ac2-ac9d-46a2-ac58-acd81570ce0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:07.333221 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d38ac2-ac9d-46a2-ac58-acd81570ce0e None admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:07.333504 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d38ac2-ac9d-46a2-ac58-acd81570ce0e None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:07.333775 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d38ac2-ac9d-46a2-ac58-acd81570ce0e None admin] PATH_INFO: `/v3/projects/0781f36cb6fd4f61b57eb625b8420b87/users/c2fd402ae6f74c89a66e314d5013e78e/roles/092036ee46ac427baf4d56468575f697` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:07.335780 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43d38ac2-ac9d-46a2-ac58-acd81570ce0e None admin] RBAC: Authorizing `identity:create_grant(project_id=0781f36cb6fd4f61b57eb625b8420b87, user_id=c2fd402ae6f74c89a66e314d5013e78e, role_id=092036ee46ac427baf4d56468575f697)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:07.337201 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43d38ac2-ac9d-46a2-ac58-acd81570ce0e None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:07.350238 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 529/1076] 10.222.0.33 () {58 vars in 1439 bytes} [Sun Oct 9 00:44:07 2022] PUT /identity/v3/projects/0781f36cb6fd4f61b57eb625b8420b87/users/c2fd402ae6f74c89a66e314d5013e78e/roles/092036ee46ac427baf4d56468575f697 => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 09 00:44:07.351750 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d38ac2-ac9d-46a2-ac58-acd81570ce0e None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:07.397847 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:07.426938 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4a40137-c7f1-451a-8ece-baadda59505a None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:07.467460 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:07.469911 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b74193b4-c4f0-4898-af76-11fea126a754 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:07.470540 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b74193b4-c4f0-4898-af76-11fea126a754 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:07.470654 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d38ac2-ac9d-46a2-ac58-acd81570ce0e None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:07.470940 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b74193b4-c4f0-4898-af76-11fea126a754 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:07.471257 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b74193b4-c4f0-4898-af76-11fea126a754 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:07.513487 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f60d4766-2593-4f8a-9509-ef662093d8cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:07.514134 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f60d4766-2593-4f8a-9509-ef662093d8cc None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:07.514468 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f60d4766-2593-4f8a-9509-ef662093d8cc None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:07.514751 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f60d4766-2593-4f8a-9509-ef662093d8cc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:07.528674 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f60d4766-2593-4f8a-9509-ef662093d8cc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:07.530398 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f60d4766-2593-4f8a-9509-ef662093d8cc None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:07.546054 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-f60d4766-2593-4f8a-9509-ef662093d8cc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:44:07.546054 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:44:07.546054 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:44:07.546054 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:44:07.546054 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:44:07.546054 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:44:07.546054 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:44:07.546054 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:44:07.546054 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:44:07.546054 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:44:07.546054 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:44:07.546054 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:44:07.546054 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:44:07.546054 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:44:07.546054 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:44:07.546054 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:44:07.547217 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:44:07.547217 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:44:07.547217 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:44:07.547217 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:44:07.547217 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:44:07.547217 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:44:07.547217 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:44:07.547217 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:44:07.547217 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:44:07.547217 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:44:07.547217 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:44:07.547217 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:44:07.547217 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:44:07.547217 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:44:07.547217 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:44:07.547217 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:44:07.547217 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:44:07.548255 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:44:07.548255 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:44:07.548255 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:44:07.548255 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 530/1077] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:44:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:44:07.554512 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f60d4766-2593-4f8a-9509-ef662093d8cc None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:07.555919 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b74193b4-c4f0-4898-af76-11fea126a754 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:07.557548 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b74193b4-c4f0-4898-af76-11fea126a754 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:07.641903 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:07.657905 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f60d4766-2593-4f8a-9509-ef662093d8cc None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:07.671628 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 548/1078] 10.222.0.33 () {58 vars in 1282 bytes} [Sun Oct 9 00:44:07 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 358 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 09 00:44:07.674088 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b74193b4-c4f0-4898-af76-11fea126a754 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:07.693657 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4286d64-6df3-481b-9432-515eb7270d8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:07.694760 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4286d64-6df3-481b-9432-515eb7270d8c None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:07.695043 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4286d64-6df3-481b-9432-515eb7270d8c None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:07.695312 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4286d64-6df3-481b-9432-515eb7270d8c None admin] PATH_INFO: `/v3/domains` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:07.696290 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4286d64-6df3-481b-9432-515eb7270d8c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:44:07.696290 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4286d64-6df3-481b-9432-515eb7270d8c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:07.697626 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4286d64-6df3-481b-9432-515eb7270d8c None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:07.711634 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 531/1079] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:44:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:44:07.716208 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4286d64-6df3-481b-9432-515eb7270d8c None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:07.731285 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:07.779859 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b74193b4-c4f0-4898-af76-11fea126a754 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:07.815703 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb93efb8-488d-4688-b5ca-111190a2f68b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:07.816266 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb93efb8-488d-4688-b5ca-111190a2f68b None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:07.816507 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb93efb8-488d-4688-b5ca-111190a2f68b None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:07.816777 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb93efb8-488d-4688-b5ca-111190a2f68b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:07.824099 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4286d64-6df3-481b-9432-515eb7270d8c None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:07.829930 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb93efb8-488d-4688-b5ca-111190a2f68b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:07.831479 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb93efb8-488d-4688-b5ca-111190a2f68b None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:07.848826 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-bb93efb8-488d-4688-b5ca-111190a2f68b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:44:07.848826 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:44:07.848826 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:44:07.848826 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:44:07.848826 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:44:07.848826 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:44:07.848826 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:44:07.848826 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:44:07.848826 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:44:07.848826 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:44:07.848826 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:44:07.848826 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:44:07.848826 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:44:07.848826 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:44:07.848826 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:44:07.848826 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:44:07.850340 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:44:07.850340 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:44:07.850340 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:44:07.850340 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:44:07.850340 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:44:07.850340 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:44:07.850340 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:44:07.850340 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:44:07.850340 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:44:07.850340 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:44:07.850340 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:44:07.850340 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:44:07.850340 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:44:07.850340 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:44:07.850340 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:44:07.850340 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:44:07.850340 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:44:07.852061 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:44:07.852061 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:44:07.852061 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:44:07.855463 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 549/1080] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:44:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:44:07.856536 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb93efb8-488d-4688-b5ca-111190a2f68b None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:07.863647 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec4f39aa-079b-4d54-adb4-6aafb5473533 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:07.864365 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec4f39aa-079b-4d54-adb4-6aafb5473533 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:07.864652 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec4f39aa-079b-4d54-adb4-6aafb5473533 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:07.864925 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec4f39aa-079b-4d54-adb4-6aafb5473533 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:07.871928 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec4f39aa-079b-4d54-adb4-6aafb5473533 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:07.873798 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec4f39aa-079b-4d54-adb4-6aafb5473533 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:07.884835 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-ec4f39aa-079b-4d54-adb4-6aafb5473533 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 09 00:44:07.884835 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:44:07.884835 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:44:07.884835 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:44:07.884835 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:44:07.884835 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:44:07.884835 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:44:07.884835 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:44:07.884835 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:44:07.884835 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:44:07.884835 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:44:07.884835 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:44:07.884835 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 09 00:44:07.884835 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 09 00:44:07.884835 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 09 00:44:07.885704 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 09 00:44:07.885704 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:44:07.885704 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:44:07.885704 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:44:07.885704 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:44:07.885704 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:44:07.885704 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:44:07.885704 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:44:07.885704 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:44:07.885704 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:44:07.885704 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:44:07.885704 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:44:07.885704 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:44:07.885704 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:44:07.885704 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:44:07.885704 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:44:07.885704 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:44:07.891505 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 09 00:44:07.891505 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 09 00:44:07.891505 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 09 00:44:07.891505 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 09 00:44:07.891505 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 09 00:44:07.891505 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 09 00:44:07.891505 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 09 00:44:07.891505 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:44:07.891505 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 532/1081] 10.222.0.33 () {56 vars in 1111 bytes} [Sun Oct 9 00:44:07 2022] GET /identity/v3/projects/service => generated 88 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 09 00:44:07.892021 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec4f39aa-079b-4d54-adb4-6aafb5473533 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:07.893592 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:07.968276 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:07.968831 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb93efb8-488d-4688-b5ca-111190a2f68b None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:08.001704 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec4f39aa-079b-4d54-adb4-6aafb5473533 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:08.003827 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfa6fede-3038-49ee-ae51-1f4cc498cb83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:08.005203 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa6fede-3038-49ee-ae51-1f4cc498cb83 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:08.005482 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa6fede-3038-49ee-ae51-1f4cc498cb83 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:08.005752 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa6fede-3038-49ee-ae51-1f4cc498cb83 None admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:08.006382 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfa6fede-3038-49ee-ae51-1f4cc498cb83 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:44:08.007234 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfa6fede-3038-49ee-ae51-1f4cc498cb83 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:08.008653 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfa6fede-3038-49ee-ae51-1f4cc498cb83 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:08.026262 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 550/1082] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:44:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:44:08.051175 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19b5a84a-2ee0-449a-b705-c2da0475f719 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:08.051837 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19b5a84a-2ee0-449a-b705-c2da0475f719 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:08.052119 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19b5a84a-2ee0-449a-b705-c2da0475f719 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:08.052397 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19b5a84a-2ee0-449a-b705-c2da0475f719 None admin] PATH_INFO: `/v3/projects` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:08.053220 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19b5a84a-2ee0-449a-b705-c2da0475f719 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:44:08.053559 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19b5a84a-2ee0-449a-b705-c2da0475f719 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:08.055045 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19b5a84a-2ee0-449a-b705-c2da0475f719 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:08.068912 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 533/1083] 10.222.0.33 () {56 vars in 1148 bytes} [Sun Oct 9 00:44:07 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:44:08.076659 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfa6fede-3038-49ee-ae51-1f4cc498cb83 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:08.174998 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:08.190117 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfa6fede-3038-49ee-ae51-1f4cc498cb83 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:08.226806 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0094193a-13d1-4071-999b-99ccb2bc3f13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:08.227484 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0094193a-13d1-4071-999b-99ccb2bc3f13 None admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:08.227855 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0094193a-13d1-4071-999b-99ccb2bc3f13 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:08.228128 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0094193a-13d1-4071-999b-99ccb2bc3f13 None admin] PATH_INFO: `/v3/projects/013c83476b1340fbb051061833c4a9c6/users/e0a5d95969e74157bb2d2a57207e1651/roles/fddb7718710e48bb8034bd9390b820a2` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:08.230372 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0094193a-13d1-4071-999b-99ccb2bc3f13 None admin] RBAC: Authorizing `identity:create_grant(project_id=013c83476b1340fbb051061833c4a9c6, user_id=e0a5d95969e74157bb2d2a57207e1651, role_id=fddb7718710e48bb8034bd9390b820a2)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:08.231895 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0094193a-13d1-4071-999b-99ccb2bc3f13 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:08.248439 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 551/1084] 10.222.0.33 () {58 vars in 1439 bytes} [Sun Oct 9 00:44:08 2022] PUT /identity/v3/projects/013c83476b1340fbb051061833c4a9c6/users/e0a5d95969e74157bb2d2a57207e1651/roles/fddb7718710e48bb8034bd9390b820a2 => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:44:09.503818 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6b1f78e-82f8-4a26-b374-e9ac85924d8a None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:09.504372 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6b1f78e-82f8-4a26-b374-e9ac85924d8a None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:09.504372 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6b1f78e-82f8-4a26-b374-e9ac85924d8a None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:09.505716 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 534/1085] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:44:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:44:09.511161 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b13e529-2781-4ab2-acb1-a71fd2aa9205 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:09.511575 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b13e529-2781-4ab2-acb1-a71fd2aa9205 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:09.512018 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b13e529-2781-4ab2-acb1-a71fd2aa9205 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:09.529761 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-1b13e529-2781-4ab2-acb1-a71fd2aa9205 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:44:09.530926 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-1b13e529-2781-4ab2-acb1-a71fd2aa9205 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:44:09.596631 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 552/1086] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:44:09 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 09 00:44:09.868107 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6b1f78e-82f8-4a26-b374-e9ac85924d8a None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:09.968573 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6b1f78e-82f8-4a26-b374-e9ac85924d8a None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:09.996680 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:10.001850 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c3f8441-7f29-4a2d-91af-994aad6bc8ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:10.002475 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c3f8441-7f29-4a2d-91af-994aad6bc8ba None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:10.002748 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c3f8441-7f29-4a2d-91af-994aad6bc8ba None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:10.003017 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c3f8441-7f29-4a2d-91af-994aad6bc8ba None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:10.012788 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c3f8441-7f29-4a2d-91af-994aad6bc8ba None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:44:10.017484 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-1c3f8441-7f29-4a2d-91af-994aad6bc8ba None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 09 00:44:10.017484 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:44:10.017484 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:44:10.017484 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:44:10.017484 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:44:10.017484 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:44:10.017484 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:44:10.017484 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:44:10.017484 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:44:10.017484 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:44:10.017484 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:44:10.017484 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:44:10.017484 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:44:10.017484 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:44:10.017484 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:44:10.017484 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:44:10.018640 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:44:10.018640 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:44:10.018640 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:44:10.018640 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:44:10.018640 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:44:10.018640 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:44:10.018640 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:44:10.018640 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:44:10.018640 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:44:10.018640 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:44:10.018640 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:44:10.018640 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:44:10.018640 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:44:10.018640 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:44:10.018640 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:44:10.018640 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:44:10.019575 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:44:10.019575 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:44:10.019575 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:44:10.019575 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:44:10.019575 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:44:10.019575 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:44:10.019575 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:44:10.019575 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:44:10.019575 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:44:10.019575 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:44:10.019575 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 09 00:44:10.019575 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:44:10.019575 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 535/1087] 10.222.0.33 () {56 vars in 1096 bytes} [Sun Oct 9 00:44:09 2022] GET /identity/v3/roles/admin => generated 83 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:44:10.025052 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b13e529-2781-4ab2-acb1-a71fd2aa9205 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:10.103317 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:10.127173 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b13e529-2781-4ab2-acb1-a71fd2aa9205 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:10.161398 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad152f97-71ac-4c3b-8de5-e0f30ec0afa6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:10.161990 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad152f97-71ac-4c3b-8de5-e0f30ec0afa6 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:10.162272 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad152f97-71ac-4c3b-8de5-e0f30ec0afa6 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:10.162546 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad152f97-71ac-4c3b-8de5-e0f30ec0afa6 None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:10.163775 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad152f97-71ac-4c3b-8de5-e0f30ec0afa6 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:44:10.164078 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad152f97-71ac-4c3b-8de5-e0f30ec0afa6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:10.165688 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad152f97-71ac-4c3b-8de5-e0f30ec0afa6 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:10.173443 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 553/1088] 10.222.0.33 () {56 vars in 1099 bytes} [Sun Oct 9 00:44:10 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:44:10.180523 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c3f8441-7f29-4a2d-91af-994aad6bc8ba None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:10.278549 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c3f8441-7f29-4a2d-91af-994aad6bc8ba None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:10.316020 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e7f932-0977-4629-8f14-c8a1772109d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:10.316647 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e7f932-0977-4629-8f14-c8a1772109d8 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:10.316927 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e7f932-0977-4629-8f14-c8a1772109d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:10.317201 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e7f932-0977-4629-8f14-c8a1772109d8 None admin] PATH_INFO: `/v3/users/c2fd402ae6f74c89a66e314d5013e78e` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:10.318323 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44e7f932-0977-4629-8f14-c8a1772109d8 None admin] RBAC: Authorizing `identity:get_user(user_id=c2fd402ae6f74c89a66e314d5013e78e)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:10.319834 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44e7f932-0977-4629-8f14-c8a1772109d8 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:10.321411 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 536/1089] 10.222.0.33 () {56 vars in 1177 bytes} [Sun Oct 9 00:44:10 2022] GET /identity/v3/users/c2fd402ae6f74c89a66e314d5013e78e => generated 285 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:44:10.327175 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad152f97-71ac-4c3b-8de5-e0f30ec0afa6 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:10.398699 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10af48ee-cb94-47e4-8bd2-74f797dfb5a9 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:10.398900 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10af48ee-cb94-47e4-8bd2-74f797dfb5a9 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:10.399159 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10af48ee-cb94-47e4-8bd2-74f797dfb5a9 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:10.400502 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 537/1090] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:44:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 09 00:44:10.405311 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e2078c-f883-4a6d-a2e5-6952225b48dc None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:10.405591 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e2078c-f883-4a6d-a2e5-6952225b48dc None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:10.405861 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e2078c-f883-4a6d-a2e5-6952225b48dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:10.424897 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-78e2078c-f883-4a6d-a2e5-6952225b48dc None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:44:10.426080 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-78e2078c-f883-4a6d-a2e5-6952225b48dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:44:10.426691 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad152f97-71ac-4c3b-8de5-e0f30ec0afa6 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:10.459565 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06163b10-edba-4bdd-9b2c-d2eca4b7529e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:10.460217 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06163b10-edba-4bdd-9b2c-d2eca4b7529e None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:10.460493 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06163b10-edba-4bdd-9b2c-d2eca4b7529e None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:10.460763 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06163b10-edba-4bdd-9b2c-d2eca4b7529e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:10.471245 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:10.481283 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:10.493842 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 538/1091] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:44:10 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 09 00:44:10.528480 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06163b10-edba-4bdd-9b2c-d2eca4b7529e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:10.529916 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06163b10-edba-4bdd-9b2c-d2eca4b7529e None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:10.624128 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 554/1092] 10.222.0.33 () {58 vars in 1282 bytes} [Sun Oct 9 00:44:10 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 297 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 09 00:44:10.630086 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78e2078c-f883-4a6d-a2e5-6952225b48dc None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:10.724952 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78e2078c-f883-4a6d-a2e5-6952225b48dc None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:10.757694 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21887c97-a4c6-4e08-a547-62d79df16b6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:10.758308 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21887c97-a4c6-4e08-a547-62d79df16b6f None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:10.758584 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21887c97-a4c6-4e08-a547-62d79df16b6f None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:10.758851 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21887c97-a4c6-4e08-a547-62d79df16b6f None admin] PATH_INFO: `/v3/projects/0781f36cb6fd4f61b57eb625b8420b87` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:10.760010 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21887c97-a4c6-4e08-a547-62d79df16b6f None admin] RBAC: Authorizing `identity:get_project(project_id=0781f36cb6fd4f61b57eb625b8420b87)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:10.761500 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21887c97-a4c6-4e08-a547-62d79df16b6f None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:10.764163 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 539/1093] 10.222.0.33 () {56 vars in 1186 bytes} [Sun Oct 9 00:44:10 2022] GET /identity/v3/projects/0781f36cb6fd4f61b57eb625b8420b87 => generated 311 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:44:10.765115 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06163b10-edba-4bdd-9b2c-d2eca4b7529e None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:10.768889 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21887c97-a4c6-4e08-a547-62d79df16b6f None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:10.807007 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:10.826321 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:10.865789 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06163b10-edba-4bdd-9b2c-d2eca4b7529e None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:10.868136 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21887c97-a4c6-4e08-a547-62d79df16b6f None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:10.900988 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f46d11c-7992-4e39-ac56-c3f9b40505da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:10.901609 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f46d11c-7992-4e39-ac56-c3f9b40505da None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:10.901887 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f46d11c-7992-4e39-ac56-c3f9b40505da None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:10.902157 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f46d11c-7992-4e39-ac56-c3f9b40505da None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:10.902746 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ada5f12-28e3-4c40-a9ed-f33d035bdb76 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:10.903336 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ada5f12-28e3-4c40-a9ed-f33d035bdb76 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:10.903643 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ada5f12-28e3-4c40-a9ed-f33d035bdb76 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:10.904082 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ada5f12-28e3-4c40-a9ed-f33d035bdb76 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:10.904793 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ada5f12-28e3-4c40-a9ed-f33d035bdb76 None admin] RBAC: Adding query filter params (role.id=092036ee46ac427baf4d56468575f697, scope.project.id=0781f36cb6fd4f61b57eb625b8420b87, user.id=c2fd402ae6f74c89a66e314d5013e78e) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:44:10.905090 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ada5f12-28e3-4c40-a9ed-f33d035bdb76 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:10.906489 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ada5f12-28e3-4c40-a9ed-f33d035bdb76 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:10.912072 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 540/1094] 10.222.0.33 () {56 vars in 1374 bytes} [Sun Oct 9 00:44:10 2022] GET /identity/v3/role_assignments?user.id=c2fd402ae6f74c89a66e314d5013e78e&scope.project.id=0781f36cb6fd4f61b57eb625b8420b87&role.id=092036ee46ac427baf4d56468575f697 => generated 608 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:44:10.912256 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f46d11c-7992-4e39-ac56-c3f9b40505da None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:44:10.918208 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-4f46d11c-7992-4e39-ac56-c3f9b40505da None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 09 00:44:10.918208 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:44:10.918208 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:44:10.918208 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:44:10.918208 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:44:10.918208 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:44:10.918208 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:44:10.918208 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:44:10.918208 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:44:10.918208 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:44:10.918208 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:44:10.918208 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:44:10.918208 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:44:10.918208 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:44:10.918208 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:44:10.918208 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:44:10.919329 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:44:10.919329 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:44:10.919329 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:44:10.919329 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:44:10.919329 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:44:10.919329 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:44:10.919329 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:44:10.919329 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:44:10.919329 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:44:10.919329 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:44:10.919329 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:44:10.919329 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:44:10.919329 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:44:10.919329 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:44:10.919329 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:44:10.919329 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:44:10.920206 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:44:10.920206 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:44:10.920206 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:44:10.920206 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:44:10.920206 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:44:10.920206 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:44:10.920206 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:44:10.920206 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:44:10.920206 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:44:10.920206 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:44:10.920206 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 09 00:44:10.920206 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:44:10.920206 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 555/1095] 10.222.0.33 () {56 vars in 1120 bytes} [Sun Oct 9 00:44:10 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 09 00:44:10.926069 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ada5f12-28e3-4c40-a9ed-f33d035bdb76 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:11.029992 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ada5f12-28e3-4c40-a9ed-f33d035bdb76 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:11.053618 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:11.065812 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907138c1-3183-4584-b311-fb1ea6298ae9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:11.066425 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907138c1-3183-4584-b311-fb1ea6298ae9 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:11.066703 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907138c1-3183-4584-b311-fb1ea6298ae9 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:11.066972 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907138c1-3183-4584-b311-fb1ea6298ae9 None admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:11.067662 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-907138c1-3183-4584-b311-fb1ea6298ae9 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:44:11.067966 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-907138c1-3183-4584-b311-fb1ea6298ae9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:11.069861 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-907138c1-3183-4584-b311-fb1ea6298ae9 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:11.081668 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 541/1096] 10.222.0.33 () {56 vars in 1115 bytes} [Sun Oct 9 00:44:10 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:44:11.088674 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f46d11c-7992-4e39-ac56-c3f9b40505da None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:11.186661 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:11.189789 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f46d11c-7992-4e39-ac56-c3f9b40505da None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:11.223562 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ce3536c-845e-4d3c-8766-1388e8699b1f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:11.224207 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ce3536c-845e-4d3c-8766-1388e8699b1f None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:11.224490 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ce3536c-845e-4d3c-8766-1388e8699b1f None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:11.224765 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ce3536c-845e-4d3c-8766-1388e8699b1f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:11.234995 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ce3536c-845e-4d3c-8766-1388e8699b1f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:11.236454 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ce3536c-845e-4d3c-8766-1388e8699b1f None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:11.251207 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-9ce3536c-845e-4d3c-8766-1388e8699b1f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:44:11.251207 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:44:11.251207 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:44:11.251207 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:44:11.251207 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:44:11.251207 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:44:11.251207 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:44:11.251207 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:44:11.251207 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:44:11.251207 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:44:11.251207 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:44:11.251207 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:44:11.251207 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:44:11.251207 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:44:11.251207 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:44:11.251207 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:44:11.252292 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:44:11.252292 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:44:11.252292 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:44:11.252292 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:44:11.252292 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:44:11.252292 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:44:11.252292 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:44:11.252292 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:44:11.252292 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:44:11.252292 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:44:11.252292 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:44:11.252292 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:44:11.252292 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:44:11.252292 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:44:11.252292 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:44:11.252292 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:44:11.252292 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:44:11.253222 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:44:11.253222 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:44:11.253222 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:44:11.253222 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 556/1097] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:44:11 2022] GET /identity/v3/domains/Default => generated 87 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:44:11.258226 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907138c1-3183-4584-b311-fb1ea6298ae9 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:11.355534 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907138c1-3183-4584-b311-fb1ea6298ae9 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:11.387552 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96e0b110-c5bf-4892-8885-2782c780cb2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:11.388175 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e0b110-c5bf-4892-8885-2782c780cb2e None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:11.388458 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e0b110-c5bf-4892-8885-2782c780cb2e None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:11.388730 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e0b110-c5bf-4892-8885-2782c780cb2e None admin] PATH_INFO: `/v3/domains` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:11.389357 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96e0b110-c5bf-4892-8885-2782c780cb2e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:44:11.389658 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96e0b110-c5bf-4892-8885-2782c780cb2e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:11.391009 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96e0b110-c5bf-4892-8885-2782c780cb2e None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:11.402976 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 542/1098] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:44:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:44:11.408679 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ce3536c-845e-4d3c-8766-1388e8699b1f None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:11.507063 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ce3536c-845e-4d3c-8766-1388e8699b1f None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:11.540077 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6610f841-3e7a-4bf6-898a-db60a215ecc4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:11.540693 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6610f841-3e7a-4bf6-898a-db60a215ecc4 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:11.540969 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6610f841-3e7a-4bf6-898a-db60a215ecc4 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:11.541239 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6610f841-3e7a-4bf6-898a-db60a215ecc4 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:11.548336 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6610f841-3e7a-4bf6-898a-db60a215ecc4 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:11.549764 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6610f841-3e7a-4bf6-898a-db60a215ecc4 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:11.560685 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-6610f841-3e7a-4bf6-898a-db60a215ecc4 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 09 00:44:11.560685 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:44:11.560685 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:44:11.560685 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:44:11.560685 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:44:11.560685 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:44:11.560685 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:44:11.560685 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:44:11.560685 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:44:11.560685 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:44:11.560685 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:44:11.560685 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:44:11.560685 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 09 00:44:11.560685 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 09 00:44:11.560685 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 09 00:44:11.560685 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 09 00:44:11.561711 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:44:11.561711 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:44:11.561711 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 09 00:44:11.561711 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 09 00:44:11.561711 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 09 00:44:11.561711 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 09 00:44:11.561711 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:44:11.561711 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:44:11.561711 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:44:11.561711 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:44:11.561711 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:44:11.561711 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:44:11.561711 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:44:11.561711 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:44:11.561711 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:44:11.561711 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:44:11.561711 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:44:11.561711 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:44:11.562681 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:44:11.562681 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:44:11.562681 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 09 00:44:11.562681 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 09 00:44:11.562681 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 09 00:44:11.562681 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 09 00:44:11.562681 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 09 00:44:11.562681 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 09 00:44:11.562681 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 09 00:44:11.562681 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:44:11.562681 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 557/1099] 10.222.0.33 () {56 vars in 1093 bytes} [Sun Oct 9 00:44:11 2022] GET /identity/v3/users/nova => generated 82 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:44:11.567482 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96e0b110-c5bf-4892-8885-2782c780cb2e None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:11.612674 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:11.666462 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96e0b110-c5bf-4892-8885-2782c780cb2e None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:11.700620 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdbc1f5c-7f6a-475b-8b36-bb4ff40adec5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:11.701216 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdbc1f5c-7f6a-475b-8b36-bb4ff40adec5 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:11.701494 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdbc1f5c-7f6a-475b-8b36-bb4ff40adec5 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:11.701765 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdbc1f5c-7f6a-475b-8b36-bb4ff40adec5 None admin] PATH_INFO: `/v3/users` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:11.702479 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdbc1f5c-7f6a-475b-8b36-bb4ff40adec5 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:44:11.702791 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdbc1f5c-7f6a-475b-8b36-bb4ff40adec5 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:11.704237 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdbc1f5c-7f6a-475b-8b36-bb4ff40adec5 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:11.721309 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 543/1100] 10.222.0.33 () {56 vars in 1133 bytes} [Sun Oct 9 00:44:11 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:44:11.727097 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6610f841-3e7a-4bf6-898a-db60a215ecc4 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:11.742043 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:11.827750 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6610f841-3e7a-4bf6-898a-db60a215ecc4 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:11.860378 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef0a9406-9587-477c-b6a4-b3a559895fda None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:11.860966 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef0a9406-9587-477c-b6a4-b3a559895fda None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:11.861254 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef0a9406-9587-477c-b6a4-b3a559895fda None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:11.861527 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef0a9406-9587-477c-b6a4-b3a559895fda None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:11.926799 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef0a9406-9587-477c-b6a4-b3a559895fda None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:11.928267 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef0a9406-9587-477c-b6a4-b3a559895fda None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:11.941230 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:12.024171 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 558/1101] 10.222.0.33 () {58 vars in 1282 bytes} [Sun Oct 9 00:44:11 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 297 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 09 00:44:12.029895 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdbc1f5c-7f6a-475b-8b36-bb4ff40adec5 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:12.129618 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdbc1f5c-7f6a-475b-8b36-bb4ff40adec5 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:12.131136 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:12.163518 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3f29c8a-831d-47af-98b4-6c52f65f4e48 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:12.164165 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3f29c8a-831d-47af-98b4-6c52f65f4e48 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:12.164698 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3f29c8a-831d-47af-98b4-6c52f65f4e48 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:12.164965 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3f29c8a-831d-47af-98b4-6c52f65f4e48 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:12.174543 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3f29c8a-831d-47af-98b4-6c52f65f4e48 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:12.175996 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3f29c8a-831d-47af-98b4-6c52f65f4e48 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:12.190570 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-f3f29c8a-831d-47af-98b4-6c52f65f4e48 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:44:12.190570 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:44:12.190570 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:44:12.190570 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:44:12.190570 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:44:12.190570 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:44:12.190570 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:44:12.190570 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:44:12.190570 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:44:12.190570 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:44:12.190570 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:44:12.190570 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:44:12.190570 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:44:12.190570 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:44:12.190570 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:44:12.190570 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:44:12.196920 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:44:12.196920 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:44:12.196920 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:44:12.196920 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:44:12.196920 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:44:12.196920 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:44:12.196920 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:44:12.196920 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:44:12.196920 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:44:12.196920 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:44:12.196920 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:44:12.196920 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:44:12.196920 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:44:12.196920 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:44:12.196920 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:44:12.196920 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:44:12.196920 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:44:12.198056 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:44:12.198056 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:44:12.198056 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:44:12.198056 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 544/1102] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:44:12 2022] GET /identity/v3/domains/Default => generated 87 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:44:12.199708 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef0a9406-9587-477c-b6a4-b3a559895fda None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:12.296104 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef0a9406-9587-477c-b6a4-b3a559895fda None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:12.309475 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:12.331305 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21a08dc9-e806-40d7-99e2-fd5ef1936f3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:12.331925 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a08dc9-e806-40d7-99e2-fd5ef1936f3f None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:12.332204 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a08dc9-e806-40d7-99e2-fd5ef1936f3f None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:12.332470 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a08dc9-e806-40d7-99e2-fd5ef1936f3f None admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:12.333089 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21a08dc9-e806-40d7-99e2-fd5ef1936f3f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:44:12.333385 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21a08dc9-e806-40d7-99e2-fd5ef1936f3f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:12.334909 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21a08dc9-e806-40d7-99e2-fd5ef1936f3f None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:12.345808 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 559/1103] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:44:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:44:12.351554 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3f29c8a-831d-47af-98b4-6c52f65f4e48 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:12.447242 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3f29c8a-831d-47af-98b4-6c52f65f4e48 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:12.479311 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bda36477-fd2c-4167-b623-49fbadd5b169 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:12.479966 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda36477-fd2c-4167-b623-49fbadd5b169 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:12.480243 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda36477-fd2c-4167-b623-49fbadd5b169 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:12.480511 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda36477-fd2c-4167-b623-49fbadd5b169 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:12.486879 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bda36477-fd2c-4167-b623-49fbadd5b169 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:12.488324 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bda36477-fd2c-4167-b623-49fbadd5b169 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:12.499744 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-bda36477-fd2c-4167-b623-49fbadd5b169 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 09 00:44:12.499744 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:44:12.499744 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:44:12.499744 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:44:12.499744 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:44:12.499744 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:44:12.499744 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:44:12.499744 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:44:12.499744 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:44:12.499744 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:44:12.499744 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:44:12.499744 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:44:12.499744 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 09 00:44:12.499744 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 09 00:44:12.499744 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 09 00:44:12.500794 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 09 00:44:12.500794 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:44:12.500794 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:44:12.500794 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:44:12.500794 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:44:12.500794 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:44:12.500794 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:44:12.500794 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:44:12.500794 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:44:12.500794 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:44:12.500794 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:44:12.500794 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:44:12.500794 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:44:12.500794 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:44:12.500794 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:44:12.500794 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:44:12.500794 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:44:12.502033 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 09 00:44:12.502033 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 09 00:44:12.502033 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 09 00:44:12.502033 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 09 00:44:12.502033 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 09 00:44:12.502033 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 09 00:44:12.502033 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 09 00:44:12.502033 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:44:12.502033 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 545/1104] 10.222.0.33 () {56 vars in 1111 bytes} [Sun Oct 9 00:44:12 2022] GET /identity/v3/projects/service => generated 88 bytes in 150 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:44:12.506606 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21a08dc9-e806-40d7-99e2-fd5ef1936f3f None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:12.605900 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21a08dc9-e806-40d7-99e2-fd5ef1936f3f None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:12.638557 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5101f10-2d41-4cd3-9718-993a10b13edc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:12.639152 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5101f10-2d41-4cd3-9718-993a10b13edc None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:12.639428 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5101f10-2d41-4cd3-9718-993a10b13edc None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:12.639731 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5101f10-2d41-4cd3-9718-993a10b13edc None admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:12.640404 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5101f10-2d41-4cd3-9718-993a10b13edc None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:44:12.640701 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5101f10-2d41-4cd3-9718-993a10b13edc None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:12.642108 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5101f10-2d41-4cd3-9718-993a10b13edc None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:12.654665 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 560/1105] 10.222.0.33 () {56 vars in 1148 bytes} [Sun Oct 9 00:44:12 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:44:12.660820 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bda36477-fd2c-4167-b623-49fbadd5b169 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:12.677231 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:12.758591 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bda36477-fd2c-4167-b623-49fbadd5b169 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:12.791607 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24b575c7-3d1b-4685-826e-3ad327558859 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:12.792215 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b575c7-3d1b-4685-826e-3ad327558859 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:12.792489 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b575c7-3d1b-4685-826e-3ad327558859 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:12.792756 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b575c7-3d1b-4685-826e-3ad327558859 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:12.793460 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24b575c7-3d1b-4685-826e-3ad327558859 None admin] RBAC: Adding query filter params (role.id=fddb7718710e48bb8034bd9390b820a2, scope.project.id=013c83476b1340fbb051061833c4a9c6, user.id=e0a5d95969e74157bb2d2a57207e1651) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:44:12.793765 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24b575c7-3d1b-4685-826e-3ad327558859 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:12.795171 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24b575c7-3d1b-4685-826e-3ad327558859 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:12.800817 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 546/1106] 10.222.0.33 () {56 vars in 1374 bytes} [Sun Oct 9 00:44:12 2022] GET /identity/v3/role_assignments?user.id=e0a5d95969e74157bb2d2a57207e1651&scope.project.id=013c83476b1340fbb051061833c4a9c6&role.id=fddb7718710e48bb8034bd9390b820a2 => generated 608 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:44:13.064911 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf7369a-4522-4617-9afa-cd3850726702 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:13.065101 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf7369a-4522-4617-9afa-cd3850726702 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:13.065384 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf7369a-4522-4617-9afa-cd3850726702 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:13.066729 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 561/1107] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:44:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:44:13.072316 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db3121ba-ba0d-41bb-90a9-e5e2ca4cd3c6 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:13.072604 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db3121ba-ba0d-41bb-90a9-e5e2ca4cd3c6 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:13.072882 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db3121ba-ba0d-41bb-90a9-e5e2ca4cd3c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:13.091541 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-db3121ba-ba0d-41bb-90a9-e5e2ca4cd3c6 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:44:13.092758 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-db3121ba-ba0d-41bb-90a9-e5e2ca4cd3c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:44:13.158870 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 547/1108] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:44:13 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 09 00:44:13.425539 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cf7369a-4522-4617-9afa-cd3850726702 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:13.457931 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:13.528848 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cf7369a-4522-4617-9afa-cd3850726702 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:13.563142 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-075f7958-b518-4dca-9df2-df13012b4469 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:13.563798 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-075f7958-b518-4dca-9df2-df13012b4469 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:13.564083 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-075f7958-b518-4dca-9df2-df13012b4469 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:13.564358 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-075f7958-b518-4dca-9df2-df13012b4469 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:13.565467 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-075f7958-b518-4dca-9df2-df13012b4469 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:13.566907 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-075f7958-b518-4dca-9df2-df13012b4469 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:13.568513 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 562/1109] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:44:13 2022] GET /identity/v3/domains/default => generated 199 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:44:13.574305 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db3121ba-ba0d-41bb-90a9-e5e2ca4cd3c6 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:13.586224 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:13.675461 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db3121ba-ba0d-41bb-90a9-e5e2ca4cd3c6 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:13.707375 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3707b0ea-e3f8-4b26-8b17-dbd2770a4db2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:13.707964 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3707b0ea-e3f8-4b26-8b17-dbd2770a4db2 None admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:13.708214 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3707b0ea-e3f8-4b26-8b17-dbd2770a4db2 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:13.708486 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3707b0ea-e3f8-4b26-8b17-dbd2770a4db2 None admin] PATH_INFO: `/v3/users` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:13.709139 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3707b0ea-e3f8-4b26-8b17-dbd2770a4db2 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:13.711066 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3707b0ea-e3f8-4b26-8b17-dbd2770a4db2 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:13.731672 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 548/1110] 10.222.0.33 () {60 vars in 1132 bytes} [Sun Oct 9 00:44:13 2022] POST /identity/v3/users => generated 286 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:44:15.896555 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f6859ad-a9dd-49f1-b618-16b8d4686ea2 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:15.897104 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f6859ad-a9dd-49f1-b618-16b8d4686ea2 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:15.897104 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f6859ad-a9dd-49f1-b618-16b8d4686ea2 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:15.898932 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 563/1111] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:44:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:44:15.904143 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66bbbf3e-f139-4f3e-a7d7-2fe659a0285f None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:15.904524 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66bbbf3e-f139-4f3e-a7d7-2fe659a0285f None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:15.904695 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66bbbf3e-f139-4f3e-a7d7-2fe659a0285f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:15.923048 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-66bbbf3e-f139-4f3e-a7d7-2fe659a0285f None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:44:15.924260 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-66bbbf3e-f139-4f3e-a7d7-2fe659a0285f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:44:15.980581 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:15.989648 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 549/1112] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:44:15 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 09 00:44:16.257924 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f6859ad-a9dd-49f1-b618-16b8d4686ea2 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:16.352135 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:16.356367 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f6859ad-a9dd-49f1-b618-16b8d4686ea2 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:16.389406 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd66adb1-64c0-45ab-8135-64375a37f5bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:16.390015 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd66adb1-64c0-45ab-8135-64375a37f5bb None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:16.390295 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd66adb1-64c0-45ab-8135-64375a37f5bb None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:16.390567 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd66adb1-64c0-45ab-8135-64375a37f5bb None admin] PATH_INFO: `/v3/roles/7c1fdac272a34be99661077b0f1b8014` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:16.391965 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd66adb1-64c0-45ab-8135-64375a37f5bb None admin] RBAC: Authorizing `identity:get_role(role_id=7c1fdac272a34be99661077b0f1b8014)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:16.393318 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd66adb1-64c0-45ab-8135-64375a37f5bb None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:16.394608 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 564/1113] 10.222.0.33 () {56 vars in 1177 bytes} [Sun Oct 9 00:44:16 2022] GET /identity/v3/roles/7c1fdac272a34be99661077b0f1b8014 => generated 223 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:44:16.400263 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66bbbf3e-f139-4f3e-a7d7-2fe659a0285f None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:16.494034 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66bbbf3e-f139-4f3e-a7d7-2fe659a0285f None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:16.526103 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efb35bbb-7ef9-4d52-bcbc-e0693bbb396a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:16.526650 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb35bbb-7ef9-4d52-bcbc-e0693bbb396a None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:16.526928 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb35bbb-7ef9-4d52-bcbc-e0693bbb396a None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:16.527200 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb35bbb-7ef9-4d52-bcbc-e0693bbb396a None admin] PATH_INFO: `/v3/users/634bbbd36b2447d09b03a2300cff71f0` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:16.540385 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efb35bbb-7ef9-4d52-bcbc-e0693bbb396a None admin] RBAC: Authorizing `identity:get_user(user_id=634bbbd36b2447d09b03a2300cff71f0)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:16.541798 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efb35bbb-7ef9-4d52-bcbc-e0693bbb396a None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:16.543345 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 550/1114] 10.222.0.33 () {56 vars in 1177 bytes} [Sun Oct 9 00:44:16 2022] GET /identity/v3/users/634bbbd36b2447d09b03a2300cff71f0 => generated 286 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:44:16.548891 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd66adb1-64c0-45ab-8135-64375a37f5bb None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:16.644546 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd66adb1-64c0-45ab-8135-64375a37f5bb None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:16.678155 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e6966f-6ced-4be1-a454-2cb96de4b6fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:16.678792 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e6966f-6ced-4be1-a454-2cb96de4b6fe None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:16.679072 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e6966f-6ced-4be1-a454-2cb96de4b6fe None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:16.679342 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e6966f-6ced-4be1-a454-2cb96de4b6fe None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:16.712274 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:16.744407 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0e6966f-6ced-4be1-a454-2cb96de4b6fe None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:16.745848 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0e6966f-6ced-4be1-a454-2cb96de4b6fe None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:16.840265 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 565/1115] 10.222.0.33 () {58 vars in 1282 bytes} [Sun Oct 9 00:44:16 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 292 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 09 00:44:16.846030 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efb35bbb-7ef9-4d52-bcbc-e0693bbb396a None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:16.896247 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:16.949685 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efb35bbb-7ef9-4d52-bcbc-e0693bbb396a None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:16.982084 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ef033a4-5465-4570-9a70-f5064b16b4a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:16.982700 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef033a4-5465-4570-9a70-f5064b16b4a2 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:16.982978 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef033a4-5465-4570-9a70-f5064b16b4a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:16.983249 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef033a4-5465-4570-9a70-f5064b16b4a2 None admin] PATH_INFO: `/v3/projects/0781f36cb6fd4f61b57eb625b8420b87` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:16.984374 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ef033a4-5465-4570-9a70-f5064b16b4a2 None admin] RBAC: Authorizing `identity:get_project(project_id=0781f36cb6fd4f61b57eb625b8420b87)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:16.985792 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ef033a4-5465-4570-9a70-f5064b16b4a2 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:16.987386 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 551/1116] 10.222.0.33 () {56 vars in 1186 bytes} [Sun Oct 9 00:44:16 2022] GET /identity/v3/projects/0781f36cb6fd4f61b57eb625b8420b87 => generated 311 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:44:16.993540 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e6966f-6ced-4be1-a454-2cb96de4b6fe None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:17.057849 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:17.092889 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e6966f-6ced-4be1-a454-2cb96de4b6fe None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:17.125299 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fe4639a-e514-4e9a-a558-ceff9b4af29b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:17.125889 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe4639a-e514-4e9a-a558-ceff9b4af29b None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:17.126168 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe4639a-e514-4e9a-a558-ceff9b4af29b None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:17.126434 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe4639a-e514-4e9a-a558-ceff9b4af29b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:17.127141 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fe4639a-e514-4e9a-a558-ceff9b4af29b None admin] RBAC: Adding query filter params (role.id=7c1fdac272a34be99661077b0f1b8014, scope.project.id=0781f36cb6fd4f61b57eb625b8420b87, user.id=634bbbd36b2447d09b03a2300cff71f0) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:44:17.127443 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fe4639a-e514-4e9a-a558-ceff9b4af29b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:17.128888 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fe4639a-e514-4e9a-a558-ceff9b4af29b None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:17.134667 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 566/1117] 10.222.0.33 () {56 vars in 1374 bytes} [Sun Oct 9 00:44:16 2022] GET /identity/v3/role_assignments?user.id=634bbbd36b2447d09b03a2300cff71f0&scope.project.id=0781f36cb6fd4f61b57eb625b8420b87&role.id=7c1fdac272a34be99661077b0f1b8014 => generated 259 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:44:19.228556 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76aa380b-1840-45e3-a769-dcf8ec5aed1e None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:19.229077 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76aa380b-1840-45e3-a769-dcf8ec5aed1e None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:19.229183 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76aa380b-1840-45e3-a769-dcf8ec5aed1e None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:19.230429 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 552/1118] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:44:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:44:19.235428 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a48115-e508-4209-8ee7-f6f47f063945 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:19.235828 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a48115-e508-4209-8ee7-f6f47f063945 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:19.236021 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a48115-e508-4209-8ee7-f6f47f063945 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:19.254791 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-02a48115-e508-4209-8ee7-f6f47f063945 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:44:19.256005 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-02a48115-e508-4209-8ee7-f6f47f063945 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:44:19.319913 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 567/1119] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:44:19 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 09 00:44:19.587190 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76aa380b-1840-45e3-a769-dcf8ec5aed1e None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:19.684321 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76aa380b-1840-45e3-a769-dcf8ec5aed1e None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:19.704023 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:19.717905 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e187b683-5204-4804-8a03-3d7c2ead2682 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:19.718556 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e187b683-5204-4804-8a03-3d7c2ead2682 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:19.718833 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e187b683-5204-4804-8a03-3d7c2ead2682 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:19.719105 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e187b683-5204-4804-8a03-3d7c2ead2682 None admin] PATH_INFO: `/v3/roles/7c1fdac272a34be99661077b0f1b8014` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:19.720483 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e187b683-5204-4804-8a03-3d7c2ead2682 None admin] RBAC: Authorizing `identity:get_role(role_id=7c1fdac272a34be99661077b0f1b8014)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:19.721836 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e187b683-5204-4804-8a03-3d7c2ead2682 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:19.723130 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 553/1120] 10.222.0.33 () {56 vars in 1177 bytes} [Sun Oct 9 00:44:19 2022] GET /identity/v3/roles/7c1fdac272a34be99661077b0f1b8014 => generated 223 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:44:19.728791 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02a48115-e508-4209-8ee7-f6f47f063945 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:19.789410 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:19.827397 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02a48115-e508-4209-8ee7-f6f47f063945 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:19.859450 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b308d90-1675-4cc8-85ce-fb1c656d010e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:19.860037 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b308d90-1675-4cc8-85ce-fb1c656d010e None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:19.860318 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b308d90-1675-4cc8-85ce-fb1c656d010e None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:19.860590 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b308d90-1675-4cc8-85ce-fb1c656d010e None admin] PATH_INFO: `/v3/users/634bbbd36b2447d09b03a2300cff71f0` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:19.861678 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b308d90-1675-4cc8-85ce-fb1c656d010e None admin] RBAC: Authorizing `identity:get_user(user_id=634bbbd36b2447d09b03a2300cff71f0)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:19.863094 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b308d90-1675-4cc8-85ce-fb1c656d010e None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:19.864664 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 568/1121] 10.222.0.33 () {56 vars in 1177 bytes} [Sun Oct 9 00:44:19 2022] GET /identity/v3/users/634bbbd36b2447d09b03a2300cff71f0 => generated 286 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:44:19.870840 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e187b683-5204-4804-8a03-3d7c2ead2682 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:19.966588 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e187b683-5204-4804-8a03-3d7c2ead2682 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:20.000987 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8613e0f9-bbbf-40c6-bb2a-b8e3fb57d449 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:20.001602 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8613e0f9-bbbf-40c6-bb2a-b8e3fb57d449 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:20.001882 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8613e0f9-bbbf-40c6-bb2a-b8e3fb57d449 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:20.002153 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8613e0f9-bbbf-40c6-bb2a-b8e3fb57d449 None admin] PATH_INFO: `/v3/projects/0781f36cb6fd4f61b57eb625b8420b87` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:20.003237 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8613e0f9-bbbf-40c6-bb2a-b8e3fb57d449 None admin] RBAC: Authorizing `identity:get_project(project_id=0781f36cb6fd4f61b57eb625b8420b87)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:20.004705 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8613e0f9-bbbf-40c6-bb2a-b8e3fb57d449 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:20.006283 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 554/1122] 10.222.0.33 () {56 vars in 1186 bytes} [Sun Oct 9 00:44:19 2022] GET /identity/v3/projects/0781f36cb6fd4f61b57eb625b8420b87 => generated 311 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:44:20.011984 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b308d90-1675-4cc8-85ce-fb1c656d010e None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:20.110857 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b308d90-1675-4cc8-85ce-fb1c656d010e None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:20.138489 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:20.144524 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-338f6fcd-1471-499c-a5f6-21389e133286 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:20.145167 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338f6fcd-1471-499c-a5f6-21389e133286 None admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:20.145445 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338f6fcd-1471-499c-a5f6-21389e133286 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:20.145717 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338f6fcd-1471-499c-a5f6-21389e133286 None admin] PATH_INFO: `/v3/projects/0781f36cb6fd4f61b57eb625b8420b87/users/634bbbd36b2447d09b03a2300cff71f0/roles/7c1fdac272a34be99661077b0f1b8014` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:20.147869 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-338f6fcd-1471-499c-a5f6-21389e133286 None admin] RBAC: Authorizing `identity:create_grant(project_id=0781f36cb6fd4f61b57eb625b8420b87, user_id=634bbbd36b2447d09b03a2300cff71f0, role_id=7c1fdac272a34be99661077b0f1b8014)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:20.149291 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-338f6fcd-1471-499c-a5f6-21389e133286 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:20.161316 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 569/1123] 10.222.0.33 () {58 vars in 1439 bytes} [Sun Oct 9 00:44:20 2022] PUT /identity/v3/projects/0781f36cb6fd4f61b57eb625b8420b87/users/634bbbd36b2447d09b03a2300cff71f0/roles/7c1fdac272a34be99661077b0f1b8014 => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 09 00:44:22.251483 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b4df7e-19dc-4585-80cb-5efd22d6320d None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:22.252034 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b4df7e-19dc-4585-80cb-5efd22d6320d None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:22.252034 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b4df7e-19dc-4585-80cb-5efd22d6320d None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:22.253399 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 555/1124] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:44:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:44:22.258828 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-777b7f73-26ac-4a17-b489-10b98bc3128c None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:22.259221 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-777b7f73-26ac-4a17-b489-10b98bc3128c None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:22.259368 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-777b7f73-26ac-4a17-b489-10b98bc3128c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:22.276897 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-777b7f73-26ac-4a17-b489-10b98bc3128c None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:44:22.278048 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-777b7f73-26ac-4a17-b489-10b98bc3128c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:44:22.341420 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 570/1125] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:44:22 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 09 00:44:22.607429 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9b4df7e-19dc-4585-80cb-5efd22d6320d None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:22.661905 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:22.704394 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9b4df7e-19dc-4585-80cb-5efd22d6320d None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:22.736917 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d8a29e9-b45c-49fc-ba00-43e1b43988ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:22.737522 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d8a29e9-b45c-49fc-ba00-43e1b43988ce None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:22.737798 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d8a29e9-b45c-49fc-ba00-43e1b43988ce None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:22.738067 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d8a29e9-b45c-49fc-ba00-43e1b43988ce None admin] PATH_INFO: `/v3/roles/7c1fdac272a34be99661077b0f1b8014` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:22.739423 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d8a29e9-b45c-49fc-ba00-43e1b43988ce None admin] RBAC: Authorizing `identity:get_role(role_id=7c1fdac272a34be99661077b0f1b8014)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:22.740799 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d8a29e9-b45c-49fc-ba00-43e1b43988ce None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:22.742099 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 556/1126] 10.222.0.33 () {56 vars in 1177 bytes} [Sun Oct 9 00:44:22 2022] GET /identity/v3/roles/7c1fdac272a34be99661077b0f1b8014 => generated 223 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:44:22.747662 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-777b7f73-26ac-4a17-b489-10b98bc3128c None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:22.841746 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-777b7f73-26ac-4a17-b489-10b98bc3128c None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:22.861203 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:22.875228 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05b09cf2-4793-4fac-a941-7e860b8f558d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:22.875885 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b09cf2-4793-4fac-a941-7e860b8f558d None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:22.876169 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b09cf2-4793-4fac-a941-7e860b8f558d None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:22.876443 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b09cf2-4793-4fac-a941-7e860b8f558d None admin] PATH_INFO: `/v3/users/634bbbd36b2447d09b03a2300cff71f0` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:22.877543 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05b09cf2-4793-4fac-a941-7e860b8f558d None admin] RBAC: Authorizing `identity:get_user(user_id=634bbbd36b2447d09b03a2300cff71f0)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:22.878968 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05b09cf2-4793-4fac-a941-7e860b8f558d None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:22.880572 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 571/1127] 10.222.0.33 () {56 vars in 1177 bytes} [Sun Oct 9 00:44:22 2022] GET /identity/v3/users/634bbbd36b2447d09b03a2300cff71f0 => generated 286 bytes in 133 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:44:22.886145 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d8a29e9-b45c-49fc-ba00-43e1b43988ce None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:22.982193 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d8a29e9-b45c-49fc-ba00-43e1b43988ce None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:23.004148 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:23.015273 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-779c8349-e5f8-4c29-a0ac-79cd5fd90527 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:23.015891 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-779c8349-e5f8-4c29-a0ac-79cd5fd90527 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:23.016171 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-779c8349-e5f8-4c29-a0ac-79cd5fd90527 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:23.016442 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-779c8349-e5f8-4c29-a0ac-79cd5fd90527 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:23.080696 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-779c8349-e5f8-4c29-a0ac-79cd5fd90527 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:23.082129 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-779c8349-e5f8-4c29-a0ac-79cd5fd90527 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:23.175411 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 557/1128] 10.222.0.33 () {58 vars in 1282 bytes} [Sun Oct 9 00:44:22 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 290 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 09 00:44:23.181329 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05b09cf2-4793-4fac-a941-7e860b8f558d None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:23.277229 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05b09cf2-4793-4fac-a941-7e860b8f558d None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:23.309929 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e53e26a3-43cb-4487-9425-8a6e32a07f08 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:23.310540 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e53e26a3-43cb-4487-9425-8a6e32a07f08 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:23.310820 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e53e26a3-43cb-4487-9425-8a6e32a07f08 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:23.311093 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e53e26a3-43cb-4487-9425-8a6e32a07f08 None admin] PATH_INFO: `/v3/projects/0781f36cb6fd4f61b57eb625b8420b87` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:23.312207 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e53e26a3-43cb-4487-9425-8a6e32a07f08 None admin] RBAC: Authorizing `identity:get_project(project_id=0781f36cb6fd4f61b57eb625b8420b87)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:23.313620 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e53e26a3-43cb-4487-9425-8a6e32a07f08 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:23.315199 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 572/1129] 10.222.0.33 () {56 vars in 1186 bytes} [Sun Oct 9 00:44:23 2022] GET /identity/v3/projects/0781f36cb6fd4f61b57eb625b8420b87 => generated 311 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:44:23.321722 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-779c8349-e5f8-4c29-a0ac-79cd5fd90527 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:23.344753 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:23.415577 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-779c8349-e5f8-4c29-a0ac-79cd5fd90527 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:23.448015 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d081e07-611b-4617-8087-f691d0d9fe38 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:23.448541 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d081e07-611b-4617-8087-f691d0d9fe38 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:23.448817 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d081e07-611b-4617-8087-f691d0d9fe38 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:23.449085 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d081e07-611b-4617-8087-f691d0d9fe38 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:23.449786 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d081e07-611b-4617-8087-f691d0d9fe38 None admin] RBAC: Adding query filter params (role.id=7c1fdac272a34be99661077b0f1b8014, scope.project.id=0781f36cb6fd4f61b57eb625b8420b87, user.id=634bbbd36b2447d09b03a2300cff71f0) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:44:23.450087 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d081e07-611b-4617-8087-f691d0d9fe38 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:23.451485 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d081e07-611b-4617-8087-f691d0d9fe38 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:23.456911 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 558/1130] 10.222.0.33 () {56 vars in 1374 bytes} [Sun Oct 9 00:44:23 2022] GET /identity/v3/role_assignments?user.id=634bbbd36b2447d09b03a2300cff71f0&scope.project.id=0781f36cb6fd4f61b57eb625b8420b87&role.id=7c1fdac272a34be99661077b0f1b8014 => generated 608 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:44:25.574140 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c14c3ee2-b106-4c84-a61d-3637835bb5bf None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:25.574711 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c14c3ee2-b106-4c84-a61d-3637835bb5bf None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:25.574711 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c14c3ee2-b106-4c84-a61d-3637835bb5bf None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:25.576030 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 573/1131] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:44:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:44:25.581118 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24709c05-f051-4443-949f-e631e7dfaf62 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:25.581527 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24709c05-f051-4443-949f-e631e7dfaf62 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:25.581663 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24709c05-f051-4443-949f-e631e7dfaf62 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:25.599728 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-24709c05-f051-4443-949f-e631e7dfaf62 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:44:25.601457 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-24709c05-f051-4443-949f-e631e7dfaf62 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:44:25.664519 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 559/1132] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:44:25 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 09 00:44:25.931744 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c14c3ee2-b106-4c84-a61d-3637835bb5bf None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:25.987380 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:26.031100 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c14c3ee2-b106-4c84-a61d-3637835bb5bf None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:26.064502 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b7da3d-badb-4282-a36c-f1d3ffc5c09f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:26.065110 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b7da3d-badb-4282-a36c-f1d3ffc5c09f None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:26.065388 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b7da3d-badb-4282-a36c-f1d3ffc5c09f None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:26.065659 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b7da3d-badb-4282-a36c-f1d3ffc5c09f None admin] PATH_INFO: `/v3/domains/default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:26.066742 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9b7da3d-badb-4282-a36c-f1d3ffc5c09f None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:26.068200 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9b7da3d-badb-4282-a36c-f1d3ffc5c09f None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:26.069772 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 574/1133] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:44:25 2022] GET /identity/v3/domains/default => generated 199 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:44:26.075693 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24709c05-f051-4443-949f-e631e7dfaf62 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:26.093681 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:26.172680 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24709c05-f051-4443-949f-e631e7dfaf62 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:26.205390 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ed5d0ed-8e6e-4f49-be10-60b022be0c02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:26.205929 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed5d0ed-8e6e-4f49-be10-60b022be0c02 None admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:26.206208 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed5d0ed-8e6e-4f49-be10-60b022be0c02 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:26.206481 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed5d0ed-8e6e-4f49-be10-60b022be0c02 None admin] PATH_INFO: `/v3/projects` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:26.207125 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ed5d0ed-8e6e-4f49-be10-60b022be0c02 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:26.208562 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ed5d0ed-8e6e-4f49-be10-60b022be0c02 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:26.234812 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 560/1134] 10.222.0.33 () {60 vars in 1141 bytes} [Sun Oct 9 00:44:26 2022] POST /identity/v3/projects => generated 311 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:44:28.382285 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2559cbf9-aa22-4152-990d-9643bc13b469 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:28.382791 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2559cbf9-aa22-4152-990d-9643bc13b469 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:28.382791 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2559cbf9-aa22-4152-990d-9643bc13b469 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:28.384093 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 575/1135] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:44:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:44:28.389334 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4972932-0bb7-454f-88f7-82e1694ce11e None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:28.389751 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4972932-0bb7-454f-88f7-82e1694ce11e None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:28.389878 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4972932-0bb7-454f-88f7-82e1694ce11e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:28.408130 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-f4972932-0bb7-454f-88f7-82e1694ce11e None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:44:28.409297 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-f4972932-0bb7-454f-88f7-82e1694ce11e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:44:28.468973 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:28.474247 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 561/1136] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:44:28 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 09 00:44:28.740901 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2559cbf9-aa22-4152-990d-9643bc13b469 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:28.839635 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2559cbf9-aa22-4152-990d-9643bc13b469 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:28.862333 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:28.873086 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5736ace-dcc1-4575-8e65-bf1307e84b2d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:28.873694 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5736ace-dcc1-4575-8e65-bf1307e84b2d None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:28.873972 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5736ace-dcc1-4575-8e65-bf1307e84b2d None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:28.874245 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5736ace-dcc1-4575-8e65-bf1307e84b2d None admin] PATH_INFO: `/v3/domains/default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:28.875336 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5736ace-dcc1-4575-8e65-bf1307e84b2d None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:28.876779 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5736ace-dcc1-4575-8e65-bf1307e84b2d None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:28.878309 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 576/1137] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:44:28 2022] GET /identity/v3/domains/default => generated 199 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:44:28.884214 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4972932-0bb7-454f-88f7-82e1694ce11e None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:28.979829 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4972932-0bb7-454f-88f7-82e1694ce11e None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:29.013325 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad4de811-7d61-4252-9999-af46e9d6e14d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:29.013860 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad4de811-7d61-4252-9999-af46e9d6e14d None admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:29.014139 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad4de811-7d61-4252-9999-af46e9d6e14d None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:29.014412 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad4de811-7d61-4252-9999-af46e9d6e14d None admin] PATH_INFO: `/v3/users` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:29.015051 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad4de811-7d61-4252-9999-af46e9d6e14d None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:29.016485 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad4de811-7d61-4252-9999-af46e9d6e14d None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:29.039183 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 562/1138] 10.222.0.33 () {60 vars in 1132 bytes} [Sun Oct 9 00:44:28 2022] POST /identity/v3/users => generated 286 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:44:31.210538 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0afd0c92-4358-451e-be1b-2eb7108f74a6 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:31.211099 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0afd0c92-4358-451e-be1b-2eb7108f74a6 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:31.211099 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0afd0c92-4358-451e-be1b-2eb7108f74a6 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:31.212559 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 577/1139] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:44:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:44:31.217797 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f533f781-d15b-4a14-9347-56db07f6da1e None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:31.218179 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f533f781-d15b-4a14-9347-56db07f6da1e None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:31.218344 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f533f781-d15b-4a14-9347-56db07f6da1e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:31.236532 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-f533f781-d15b-4a14-9347-56db07f6da1e None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:44:31.237694 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-f533f781-d15b-4a14-9347-56db07f6da1e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:44:31.285052 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:31.302230 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 563/1140] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:44:31 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 09 00:44:31.565157 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0afd0c92-4358-451e-be1b-2eb7108f74a6 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:31.663685 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0afd0c92-4358-451e-be1b-2eb7108f74a6 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:31.697119 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01cea3f6-49ba-4bb1-8741-8a2e7a4c54e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:31.697727 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01cea3f6-49ba-4bb1-8741-8a2e7a4c54e6 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:31.698010 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01cea3f6-49ba-4bb1-8741-8a2e7a4c54e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:31.698286 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01cea3f6-49ba-4bb1-8741-8a2e7a4c54e6 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:31.707938 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01cea3f6-49ba-4bb1-8741-8a2e7a4c54e6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:44:31.712673 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-01cea3f6-49ba-4bb1-8741-8a2e7a4c54e6 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 09 00:44:31.712673 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:44:31.712673 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:44:31.712673 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:44:31.712673 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:44:31.712673 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:44:31.712673 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:44:31.712673 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:44:31.712673 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:44:31.712673 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:44:31.712673 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:44:31.712673 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:44:31.712673 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:44:31.712673 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:44:31.712673 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:44:31.712673 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:44:31.713778 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:44:31.713778 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:44:31.713778 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:44:31.713778 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:44:31.713778 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:44:31.713778 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:44:31.713778 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:44:31.713778 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:44:31.713778 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:44:31.713778 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:44:31.713778 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:44:31.713778 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:44:31.713778 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:44:31.713778 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:44:31.713778 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:44:31.713778 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:44:31.714623 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:44:31.714623 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:44:31.714623 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:44:31.714623 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:44:31.714623 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:44:31.714623 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:44:31.714623 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:44:31.714623 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:44:31.714623 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:44:31.714623 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:44:31.714623 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 09 00:44:31.714623 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:44:31.714623 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 578/1141] 10.222.0.33 () {56 vars in 1096 bytes} [Sun Oct 9 00:44:31 2022] GET /identity/v3/roles/admin => generated 83 bytes in 149 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:44:31.719435 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f533f781-d15b-4a14-9347-56db07f6da1e None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:31.814160 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f533f781-d15b-4a14-9347-56db07f6da1e None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:31.846195 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e4af7a1-cc42-40d8-afe1-9ad957fa4b77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:31.846724 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e4af7a1-cc42-40d8-afe1-9ad957fa4b77 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:31.847001 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e4af7a1-cc42-40d8-afe1-9ad957fa4b77 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:31.847270 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e4af7a1-cc42-40d8-afe1-9ad957fa4b77 None admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:31.847952 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e4af7a1-cc42-40d8-afe1-9ad957fa4b77 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:44:31.848250 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e4af7a1-cc42-40d8-afe1-9ad957fa4b77 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:31.849592 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e4af7a1-cc42-40d8-afe1-9ad957fa4b77 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:31.857235 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 564/1142] 10.222.0.33 () {56 vars in 1099 bytes} [Sun Oct 9 00:44:31 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:44:31.863003 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01cea3f6-49ba-4bb1-8741-8a2e7a4c54e6 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:31.915841 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:31.961220 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01cea3f6-49ba-4bb1-8741-8a2e7a4c54e6 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:31.994236 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aa82e6e-339c-4234-b756-0eac4c9de226 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:31.994842 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa82e6e-339c-4234-b756-0eac4c9de226 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:31.995121 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa82e6e-339c-4234-b756-0eac4c9de226 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:31.995393 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa82e6e-339c-4234-b756-0eac4c9de226 None admin] PATH_INFO: `/v3/users/3f139a8afb3844e2b5ea8581988a93ff` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:32.007913 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aa82e6e-339c-4234-b756-0eac4c9de226 None admin] RBAC: Authorizing `identity:get_user(user_id=3f139a8afb3844e2b5ea8581988a93ff)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:32.009353 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aa82e6e-339c-4234-b756-0eac4c9de226 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:32.010901 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 579/1143] 10.222.0.33 () {56 vars in 1177 bytes} [Sun Oct 9 00:44:31 2022] GET /identity/v3/users/3f139a8afb3844e2b5ea8581988a93ff => generated 286 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:44:32.016464 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e4af7a1-cc42-40d8-afe1-9ad957fa4b77 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:32.064261 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:32.119292 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e4af7a1-cc42-40d8-afe1-9ad957fa4b77 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:32.150896 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df45e588-1029-4904-9b87-52393d6b5bad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:32.151486 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df45e588-1029-4904-9b87-52393d6b5bad None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:32.151792 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df45e588-1029-4904-9b87-52393d6b5bad None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:32.152066 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df45e588-1029-4904-9b87-52393d6b5bad None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:32.216341 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df45e588-1029-4904-9b87-52393d6b5bad None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:32.217774 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df45e588-1029-4904-9b87-52393d6b5bad None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:32.262540 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:32.314427 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 565/1144] 10.222.0.33 () {58 vars in 1282 bytes} [Sun Oct 9 00:44:32 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 298 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 09 00:44:32.320354 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aa82e6e-339c-4234-b756-0eac4c9de226 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:32.417441 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aa82e6e-339c-4234-b756-0eac4c9de226 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:32.427722 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:32.450379 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3585e52c-4d5b-458a-8977-77225cb69484 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:32.450990 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3585e52c-4d5b-458a-8977-77225cb69484 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:32.451267 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3585e52c-4d5b-458a-8977-77225cb69484 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:32.451539 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3585e52c-4d5b-458a-8977-77225cb69484 None admin] PATH_INFO: `/v3/projects/ba377d3f4aa1491eb109cd1781263257` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:32.452728 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3585e52c-4d5b-458a-8977-77225cb69484 None admin] RBAC: Authorizing `identity:get_project(project_id=ba377d3f4aa1491eb109cd1781263257)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:32.454139 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3585e52c-4d5b-458a-8977-77225cb69484 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:32.455749 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 580/1145] 10.222.0.33 () {56 vars in 1186 bytes} [Sun Oct 9 00:44:32 2022] GET /identity/v3/projects/ba377d3f4aa1491eb109cd1781263257 => generated 311 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:44:32.462301 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df45e588-1029-4904-9b87-52393d6b5bad None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:32.556857 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df45e588-1029-4904-9b87-52393d6b5bad None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:32.589788 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0069386f-91cf-4584-be39-d1018265846a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:32.590373 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0069386f-91cf-4584-be39-d1018265846a None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:32.590657 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0069386f-91cf-4584-be39-d1018265846a None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:32.590931 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0069386f-91cf-4584-be39-d1018265846a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:32.591673 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0069386f-91cf-4584-be39-d1018265846a None admin] RBAC: Adding query filter params (role.id=092036ee46ac427baf4d56468575f697, scope.project.id=ba377d3f4aa1491eb109cd1781263257, user.id=3f139a8afb3844e2b5ea8581988a93ff) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:44:32.591982 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0069386f-91cf-4584-be39-d1018265846a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:32.593399 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0069386f-91cf-4584-be39-d1018265846a None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:32.598523 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 566/1146] 10.222.0.33 () {56 vars in 1374 bytes} [Sun Oct 9 00:44:32 2022] GET /identity/v3/role_assignments?user.id=3f139a8afb3844e2b5ea8581988a93ff&scope.project.id=ba377d3f4aa1491eb109cd1781263257&role.id=092036ee46ac427baf4d56468575f697 => generated 259 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:44:34.692724 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58b0a03a-1d36-4fcd-95d4-7ab4c890cd37 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:34.693288 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58b0a03a-1d36-4fcd-95d4-7ab4c890cd37 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:34.693288 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58b0a03a-1d36-4fcd-95d4-7ab4c890cd37 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:34.694601 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 581/1147] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:44:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:44:34.699712 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87001b38-30ac-4376-89e1-3fce1de24bff None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:34.700130 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87001b38-30ac-4376-89e1-3fce1de24bff None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:34.700250 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87001b38-30ac-4376-89e1-3fce1de24bff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:34.718450 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-87001b38-30ac-4376-89e1-3fce1de24bff None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:44:34.719663 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-87001b38-30ac-4376-89e1-3fce1de24bff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:44:34.727314 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:34.784229 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 567/1148] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:44:34 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 09 00:44:35.050108 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58b0a03a-1d36-4fcd-95d4-7ab4c890cd37 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:35.153828 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58b0a03a-1d36-4fcd-95d4-7ab4c890cd37 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:35.186127 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fd37c2b-bcd9-4ea7-b2b3-edafd8ac5c89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:35.186737 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd37c2b-bcd9-4ea7-b2b3-edafd8ac5c89 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:35.187015 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd37c2b-bcd9-4ea7-b2b3-edafd8ac5c89 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:35.187286 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd37c2b-bcd9-4ea7-b2b3-edafd8ac5c89 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:35.197535 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fd37c2b-bcd9-4ea7-b2b3-edafd8ac5c89 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:44:35.202223 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-2fd37c2b-bcd9-4ea7-b2b3-edafd8ac5c89 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 09 00:44:35.202223 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:44:35.202223 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:44:35.202223 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:44:35.202223 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:44:35.202223 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:44:35.202223 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:44:35.202223 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:44:35.202223 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:44:35.202223 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:44:35.202223 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:44:35.202223 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:44:35.202223 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:44:35.202223 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:44:35.202223 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:44:35.202223 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:44:35.203371 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:44:35.203371 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:44:35.203371 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:44:35.203371 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:44:35.203371 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:44:35.203371 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:44:35.203371 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:44:35.203371 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:44:35.203371 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:44:35.203371 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:44:35.203371 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:44:35.203371 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:44:35.203371 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:44:35.203371 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:44:35.203371 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:44:35.203371 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:44:35.204558 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:44:35.204558 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:44:35.204558 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:44:35.204558 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:44:35.204558 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:44:35.204558 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:44:35.204558 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:44:35.204558 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:44:35.204558 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:44:35.204558 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:44:35.204558 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 09 00:44:35.204558 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:44:35.204558 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 582/1149] 10.222.0.33 () {56 vars in 1096 bytes} [Sun Oct 9 00:44:35 2022] GET /identity/v3/roles/admin => generated 83 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:44:35.209387 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87001b38-30ac-4376-89e1-3fce1de24bff None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:35.304252 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87001b38-30ac-4376-89e1-3fce1de24bff None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:35.337076 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04126d25-547f-47a7-8c4b-21b242dc0fa8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:35.337672 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04126d25-547f-47a7-8c4b-21b242dc0fa8 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:35.337955 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04126d25-547f-47a7-8c4b-21b242dc0fa8 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:35.338231 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04126d25-547f-47a7-8c4b-21b242dc0fa8 None admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:35.338878 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04126d25-547f-47a7-8c4b-21b242dc0fa8 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:44:35.339179 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04126d25-547f-47a7-8c4b-21b242dc0fa8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:35.340571 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04126d25-547f-47a7-8c4b-21b242dc0fa8 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:35.348257 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 568/1150] 10.222.0.33 () {56 vars in 1099 bytes} [Sun Oct 9 00:44:35 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:44:35.354052 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fd37c2b-bcd9-4ea7-b2b3-edafd8ac5c89 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:35.393180 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:35.451862 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fd37c2b-bcd9-4ea7-b2b3-edafd8ac5c89 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:35.484276 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ec67307-ee62-494c-ac70-d6de24ef765f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:35.484877 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec67307-ee62-494c-ac70-d6de24ef765f None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:35.485156 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec67307-ee62-494c-ac70-d6de24ef765f None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:35.485428 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec67307-ee62-494c-ac70-d6de24ef765f None admin] PATH_INFO: `/v3/users/3f139a8afb3844e2b5ea8581988a93ff` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:35.486520 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ec67307-ee62-494c-ac70-d6de24ef765f None admin] RBAC: Authorizing `identity:get_user(user_id=3f139a8afb3844e2b5ea8581988a93ff)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:35.487973 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ec67307-ee62-494c-ac70-d6de24ef765f None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:35.489499 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 583/1151] 10.222.0.33 () {56 vars in 1177 bytes} [Sun Oct 9 00:44:35 2022] GET /identity/v3/users/3f139a8afb3844e2b5ea8581988a93ff => generated 286 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:44:35.495759 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04126d25-547f-47a7-8c4b-21b242dc0fa8 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:35.499401 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:35.592963 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04126d25-547f-47a7-8c4b-21b242dc0fa8 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:35.624921 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af86d71d-febf-4b20-9666-dea9d9fb507e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:35.625536 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af86d71d-febf-4b20-9666-dea9d9fb507e None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:35.625817 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af86d71d-febf-4b20-9666-dea9d9fb507e None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:35.626088 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af86d71d-febf-4b20-9666-dea9d9fb507e None admin] PATH_INFO: `/v3/projects/ba377d3f4aa1491eb109cd1781263257` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:35.627183 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af86d71d-febf-4b20-9666-dea9d9fb507e None admin] RBAC: Authorizing `identity:get_project(project_id=ba377d3f4aa1491eb109cd1781263257)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:35.628631 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af86d71d-febf-4b20-9666-dea9d9fb507e None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:35.630213 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 569/1152] 10.222.0.33 () {56 vars in 1186 bytes} [Sun Oct 9 00:44:35 2022] GET /identity/v3/projects/ba377d3f4aa1491eb109cd1781263257 => generated 311 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:44:35.636470 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ec67307-ee62-494c-ac70-d6de24ef765f None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:35.732493 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ec67307-ee62-494c-ac70-d6de24ef765f None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:35.739421 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:35.765861 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5734582-ce33-4746-8706-98842325b2b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:35.766494 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5734582-ce33-4746-8706-98842325b2b9 None admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:35.766774 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5734582-ce33-4746-8706-98842325b2b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:35.767049 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5734582-ce33-4746-8706-98842325b2b9 None admin] PATH_INFO: `/v3/projects/ba377d3f4aa1491eb109cd1781263257/users/3f139a8afb3844e2b5ea8581988a93ff/roles/092036ee46ac427baf4d56468575f697` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:35.768935 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5734582-ce33-4746-8706-98842325b2b9 None admin] RBAC: Authorizing `identity:create_grant(project_id=ba377d3f4aa1491eb109cd1781263257, user_id=3f139a8afb3844e2b5ea8581988a93ff, role_id=092036ee46ac427baf4d56468575f697)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:35.770346 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5734582-ce33-4746-8706-98842325b2b9 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:35.783352 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 584/1153] 10.222.0.33 () {58 vars in 1439 bytes} [Sun Oct 9 00:44:35 2022] PUT /identity/v3/projects/ba377d3f4aa1491eb109cd1781263257/users/3f139a8afb3844e2b5ea8581988a93ff/roles/092036ee46ac427baf4d56468575f697 => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:44:37.865829 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ba03cb-559b-4602-9468-02c257abd855 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:37.866383 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ba03cb-559b-4602-9468-02c257abd855 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:37.866383 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ba03cb-559b-4602-9468-02c257abd855 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:37.867719 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 570/1154] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:44:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:44:37.873427 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91eee9cb-2126-4c19-9504-63ab673b7d99 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:37.873872 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91eee9cb-2126-4c19-9504-63ab673b7d99 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:37.873975 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91eee9cb-2126-4c19-9504-63ab673b7d99 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:37.891511 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-91eee9cb-2126-4c19-9504-63ab673b7d99 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:44:37.892704 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-91eee9cb-2126-4c19-9504-63ab673b7d99 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:44:37.957046 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 585/1155] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:44:37 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 09 00:44:38.225616 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2ba03cb-559b-4602-9468-02c257abd855 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:38.298624 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:38.328980 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2ba03cb-559b-4602-9468-02c257abd855 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:38.361327 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5899c1ad-c756-45fd-826d-5135ea4ecae2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:38.361938 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5899c1ad-c756-45fd-826d-5135ea4ecae2 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:38.362214 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5899c1ad-c756-45fd-826d-5135ea4ecae2 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:38.362491 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5899c1ad-c756-45fd-826d-5135ea4ecae2 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:38.372553 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5899c1ad-c756-45fd-826d-5135ea4ecae2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:44:38.377256 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-5899c1ad-c756-45fd-826d-5135ea4ecae2 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 09 00:44:38.377256 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:44:38.377256 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:44:38.377256 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:44:38.377256 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:44:38.377256 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:44:38.377256 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:44:38.377256 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:44:38.377256 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:44:38.377256 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:44:38.377256 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:44:38.377256 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:44:38.377256 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:44:38.377256 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:44:38.377256 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:44:38.377256 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:44:38.378302 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:44:38.378302 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:44:38.378302 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:44:38.378302 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:44:38.378302 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:44:38.378302 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:44:38.378302 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:44:38.378302 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:44:38.378302 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:44:38.378302 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:44:38.378302 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:44:38.378302 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:44:38.378302 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:44:38.378302 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:44:38.378302 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:44:38.378302 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:44:38.379146 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:44:38.379146 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:44:38.379146 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:44:38.379146 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:44:38.379146 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:44:38.379146 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:44:38.379146 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:44:38.379146 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:44:38.379146 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:44:38.379146 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:44:38.379146 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 09 00:44:38.379146 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:44:38.379146 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 571/1156] 10.222.0.33 () {56 vars in 1096 bytes} [Sun Oct 9 00:44:38 2022] GET /identity/v3/roles/admin => generated 83 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:44:38.384484 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91eee9cb-2126-4c19-9504-63ab673b7d99 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:38.473802 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:38.480604 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91eee9cb-2126-4c19-9504-63ab673b7d99 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:38.514304 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55423fc9-4ee8-4789-974c-10200a4d7759 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:38.514893 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55423fc9-4ee8-4789-974c-10200a4d7759 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:38.515167 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55423fc9-4ee8-4789-974c-10200a4d7759 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:38.515432 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55423fc9-4ee8-4789-974c-10200a4d7759 None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:38.516105 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55423fc9-4ee8-4789-974c-10200a4d7759 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:44:38.516401 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55423fc9-4ee8-4789-974c-10200a4d7759 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:38.517735 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55423fc9-4ee8-4789-974c-10200a4d7759 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:38.525412 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 586/1157] 10.222.0.33 () {56 vars in 1099 bytes} [Sun Oct 9 00:44:38 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:44:38.531188 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5899c1ad-c756-45fd-826d-5135ea4ecae2 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:38.627125 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5899c1ad-c756-45fd-826d-5135ea4ecae2 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:38.655035 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:38.660017 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81dea82e-b53d-40b0-ba52-5bcddb65f3b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:38.660625 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81dea82e-b53d-40b0-ba52-5bcddb65f3b9 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:38.660904 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81dea82e-b53d-40b0-ba52-5bcddb65f3b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:38.661173 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81dea82e-b53d-40b0-ba52-5bcddb65f3b9 None admin] PATH_INFO: `/v3/users/3f139a8afb3844e2b5ea8581988a93ff` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:38.662268 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81dea82e-b53d-40b0-ba52-5bcddb65f3b9 None admin] RBAC: Authorizing `identity:get_user(user_id=3f139a8afb3844e2b5ea8581988a93ff)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:38.663722 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81dea82e-b53d-40b0-ba52-5bcddb65f3b9 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:38.665292 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 572/1158] 10.222.0.33 () {56 vars in 1177 bytes} [Sun Oct 9 00:44:38 2022] GET /identity/v3/users/3f139a8afb3844e2b5ea8581988a93ff => generated 286 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:44:38.671149 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55423fc9-4ee8-4789-974c-10200a4d7759 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:38.768066 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55423fc9-4ee8-4789-974c-10200a4d7759 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:38.800156 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a674212a-63cc-42c6-b26b-c25be9afecac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:38.800747 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a674212a-63cc-42c6-b26b-c25be9afecac None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:38.801022 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a674212a-63cc-42c6-b26b-c25be9afecac None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:38.801290 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a674212a-63cc-42c6-b26b-c25be9afecac None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:38.827432 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:38.866674 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a674212a-63cc-42c6-b26b-c25be9afecac None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:38.868158 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a674212a-63cc-42c6-b26b-c25be9afecac None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:38.961003 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 587/1159] 10.222.0.33 () {58 vars in 1282 bytes} [Sun Oct 9 00:44:38 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 290 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 09 00:44:38.967930 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81dea82e-b53d-40b0-ba52-5bcddb65f3b9 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:39.063034 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81dea82e-b53d-40b0-ba52-5bcddb65f3b9 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:39.095733 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30f53e7c-31ae-429d-a771-419cc8bba5db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:39.096358 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30f53e7c-31ae-429d-a771-419cc8bba5db None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:39.096636 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30f53e7c-31ae-429d-a771-419cc8bba5db None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:39.096913 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30f53e7c-31ae-429d-a771-419cc8bba5db None admin] PATH_INFO: `/v3/projects/ba377d3f4aa1491eb109cd1781263257` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:39.098009 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30f53e7c-31ae-429d-a771-419cc8bba5db None admin] RBAC: Authorizing `identity:get_project(project_id=ba377d3f4aa1491eb109cd1781263257)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:39.099428 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30f53e7c-31ae-429d-a771-419cc8bba5db None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:39.101067 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 573/1160] 10.222.0.33 () {56 vars in 1186 bytes} [Sun Oct 9 00:44:38 2022] GET /identity/v3/projects/ba377d3f4aa1491eb109cd1781263257 => generated 311 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:44:39.107634 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a674212a-63cc-42c6-b26b-c25be9afecac None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:39.163427 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:39.203610 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a674212a-63cc-42c6-b26b-c25be9afecac None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:39.236699 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc4b8b60-a042-4363-8428-c5eca0d7257b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:39.237281 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc4b8b60-a042-4363-8428-c5eca0d7257b None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:39.237557 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc4b8b60-a042-4363-8428-c5eca0d7257b None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:39.237826 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc4b8b60-a042-4363-8428-c5eca0d7257b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:39.238531 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc4b8b60-a042-4363-8428-c5eca0d7257b None admin] RBAC: Adding query filter params (role.id=092036ee46ac427baf4d56468575f697, scope.project.id=ba377d3f4aa1491eb109cd1781263257, user.id=3f139a8afb3844e2b5ea8581988a93ff) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:44:39.238833 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc4b8b60-a042-4363-8428-c5eca0d7257b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:39.240265 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc4b8b60-a042-4363-8428-c5eca0d7257b None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:39.246269 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 588/1161] 10.222.0.33 () {56 vars in 1374 bytes} [Sun Oct 9 00:44:39 2022] GET /identity/v3/role_assignments?user.id=3f139a8afb3844e2b5ea8581988a93ff&scope.project.id=ba377d3f4aa1491eb109cd1781263257&role.id=092036ee46ac427baf4d56468575f697 => generated 608 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:44:41.378079 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a14158-f7ce-4f26-8b5b-f5d486a2f067 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:41.378637 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a14158-f7ce-4f26-8b5b-f5d486a2f067 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:41.378637 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a14158-f7ce-4f26-8b5b-f5d486a2f067 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:41.379902 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 574/1162] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:44:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:44:41.385765 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e802ed-22ab-4f7e-9a79-1da650034cfb None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:41.386166 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e802ed-22ab-4f7e-9a79-1da650034cfb None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:41.386309 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e802ed-22ab-4f7e-9a79-1da650034cfb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:41.403958 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-76e802ed-22ab-4f7e-9a79-1da650034cfb None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:44:41.405120 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-76e802ed-22ab-4f7e-9a79-1da650034cfb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:44:41.469406 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 589/1163] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:44:41 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 09 00:44:41.737184 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8a14158-f7ce-4f26-8b5b-f5d486a2f067 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:41.802655 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:41.836075 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8a14158-f7ce-4f26-8b5b-f5d486a2f067 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:41.869001 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10b4680b-9454-4d65-8681-00ef28a36127 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:41.869593 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10b4680b-9454-4d65-8681-00ef28a36127 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:41.869879 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10b4680b-9454-4d65-8681-00ef28a36127 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:41.870149 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10b4680b-9454-4d65-8681-00ef28a36127 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:41.934727 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10b4680b-9454-4d65-8681-00ef28a36127 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:41.936179 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10b4680b-9454-4d65-8681-00ef28a36127 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:41.997423 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:42.030754 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 575/1164] 10.222.0.33 () {58 vars in 1282 bytes} [Sun Oct 9 00:44:41 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 294 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 09 00:44:42.036565 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76e802ed-22ab-4f7e-9a79-1da650034cfb None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:42.086630 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:42.132461 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76e802ed-22ab-4f7e-9a79-1da650034cfb None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:42.164705 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4eb3c53-0000-473e-baee-496b06164f5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:42.166165 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4eb3c53-0000-473e-baee-496b06164f5a None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:42.166437 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4eb3c53-0000-473e-baee-496b06164f5a None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:42.166702 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4eb3c53-0000-473e-baee-496b06164f5a None admin] PATH_INFO: `/v3/domains/swift_test` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:42.173018 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4eb3c53-0000-473e-baee-496b06164f5a None admin] RBAC: Authorizing `identity:get_domain(domain_id=swift_test)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:42.174438 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4eb3c53-0000-473e-baee-496b06164f5a None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:42.185323 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-d4eb3c53-0000-473e-baee-496b06164f5a None admin] Could not find domain: swift_test.: keystone.exception.DomainNotFound: Could not find domain: swift_test. Oct 09 00:44:42.185323 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:44:42.185323 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 712, in get_domain Oct 09 00:44:42.185323 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application project = self.driver.get_project(domain_id) Oct 09 00:44:42.185323 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 09 00:44:42.185323 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 09 00:44:42.185323 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 09 00:44:42.185323 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 09 00:44:42.185323 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: swift_test. Oct 09 00:44:42.185323 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:44:42.185323 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 09 00:44:42.185323 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:44:42.185323 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:44:42.185323 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:44:42.185323 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:44:42.185323 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:44:42.185323 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:44:42.186575 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:44:42.186575 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:44:42.186575 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:44:42.186575 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:44:42.186575 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:44:42.186575 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:44:42.186575 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:44:42.186575 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:44:42.186575 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:44:42.186575 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:44:42.186575 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:44:42.186575 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:44:42.186575 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:44:42.186575 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:44:42.186575 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:44:42.186575 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:44:42.187478 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:44:42.187478 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:44:42.187478 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:44:42.187478 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:44:42.187478 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:44:42.187478 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:44:42.187478 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:44:42.187478 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:44:42.187478 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:44:42.187478 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:44:42.187478 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 718, in get_domain Oct 09 00:44:42.187478 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:44:42.187478 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: swift_test. Oct 09 00:44:42.187478 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:44:42.187478 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 590/1165] 10.222.0.33 () {56 vars in 1117 bytes} [Sun Oct 9 00:44:42 2022] GET /identity/v3/domains/swift_test => generated 90 bytes in 150 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 09 00:44:42.192356 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10b4680b-9454-4d65-8681-00ef28a36127 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:42.286226 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10b4680b-9454-4d65-8681-00ef28a36127 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:42.322274 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61d805ec-15ed-4e5c-b171-ffbc7156bb65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:42.322861 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d805ec-15ed-4e5c-b171-ffbc7156bb65 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:42.323139 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d805ec-15ed-4e5c-b171-ffbc7156bb65 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:42.323409 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d805ec-15ed-4e5c-b171-ffbc7156bb65 None admin] PATH_INFO: `/v3/domains` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:42.324060 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61d805ec-15ed-4e5c-b171-ffbc7156bb65 None admin] RBAC: Adding query filter params (name=swift_test) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:44:42.324359 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61d805ec-15ed-4e5c-b171-ffbc7156bb65 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:42.325694 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61d805ec-15ed-4e5c-b171-ffbc7156bb65 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:42.332578 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 576/1166] 10.222.0.33 () {56 vars in 1115 bytes} [Sun Oct 9 00:44:42 2022] GET /identity/v3/domains?name=swift_test => generated 125 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:44:44.429591 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc03b0ce-f0ba-4f37-8e84-93c9d79eabd6 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:44.430118 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc03b0ce-f0ba-4f37-8e84-93c9d79eabd6 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:44.430118 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc03b0ce-f0ba-4f37-8e84-93c9d79eabd6 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:44.432024 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 591/1167] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:44:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:44:44.437122 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91380def-89db-43dc-97a0-8d0c1d1dfe19 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:44.437549 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91380def-89db-43dc-97a0-8d0c1d1dfe19 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:44.437666 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91380def-89db-43dc-97a0-8d0c1d1dfe19 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:44.455849 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-91380def-89db-43dc-97a0-8d0c1d1dfe19 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:44:44.457023 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-91380def-89db-43dc-97a0-8d0c1d1dfe19 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:44:44.482145 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:44.521523 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 577/1168] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:44:44 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 09 00:44:44.786765 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc03b0ce-f0ba-4f37-8e84-93c9d79eabd6 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:44.885562 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc03b0ce-f0ba-4f37-8e84-93c9d79eabd6 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:44.898940 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:44.920995 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aac91ac-e2ca-421e-907c-aed1f00155ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:44.921578 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aac91ac-e2ca-421e-907c-aed1f00155ea None admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:44.921854 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aac91ac-e2ca-421e-907c-aed1f00155ea None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:44.922122 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aac91ac-e2ca-421e-907c-aed1f00155ea None admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:44.922768 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aac91ac-e2ca-421e-907c-aed1f00155ea None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:44.924154 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aac91ac-e2ca-421e-907c-aed1f00155ea None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:44.939323 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 592/1169] 10.222.0.33 () {60 vars in 1138 bytes} [Sun Oct 9 00:44:44 2022] POST /identity/v3/domains => generated 267 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:44:47.120325 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1577778-9307-4e52-8754-be1e9b3e9750 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:47.120773 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1577778-9307-4e52-8754-be1e9b3e9750 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:47.120885 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1577778-9307-4e52-8754-be1e9b3e9750 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:47.122184 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 578/1170] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:44:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:44:47.127470 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-350b844e-e1ec-47d2-bad4-4bfd891b7f6a None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:47.127905 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-350b844e-e1ec-47d2-bad4-4bfd891b7f6a None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:47.128083 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-350b844e-e1ec-47d2-bad4-4bfd891b7f6a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:47.145825 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-350b844e-e1ec-47d2-bad4-4bfd891b7f6a None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:44:47.147014 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-350b844e-e1ec-47d2-bad4-4bfd891b7f6a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:44:47.212601 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 593/1171] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:44:47 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 09 00:44:47.479577 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1577778-9307-4e52-8754-be1e9b3e9750 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:47.576775 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1577778-9307-4e52-8754-be1e9b3e9750 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:47.609648 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3202227b-bef9-4e21-9e80-246f501f7335 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:47.610255 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3202227b-bef9-4e21-9e80-246f501f7335 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:47.610535 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3202227b-bef9-4e21-9e80-246f501f7335 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:47.610805 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3202227b-bef9-4e21-9e80-246f501f7335 None admin] PATH_INFO: `/v3/domains/a197d41794d649a2906072d7242baa4e` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:47.620862 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3202227b-bef9-4e21-9e80-246f501f7335 None admin] RBAC: Authorizing `identity:get_domain(domain_id=a197d41794d649a2906072d7242baa4e)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:47.622282 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3202227b-bef9-4e21-9e80-246f501f7335 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:47.623852 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 579/1172] 10.222.0.33 () {56 vars in 1183 bytes} [Sun Oct 9 00:44:47 2022] GET /identity/v3/domains/a197d41794d649a2906072d7242baa4e => generated 267 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:44:47.629502 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-350b844e-e1ec-47d2-bad4-4bfd891b7f6a None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:47.725071 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-350b844e-e1ec-47d2-bad4-4bfd891b7f6a None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:47.726497 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:47.760796 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a92a9974-d6bf-4d10-aca4-e1fdadc4b71f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:47.761331 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92a9974-d6bf-4d10-aca4-e1fdadc4b71f None admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:47.761607 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92a9974-d6bf-4d10-aca4-e1fdadc4b71f None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:47.761878 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92a9974-d6bf-4d10-aca4-e1fdadc4b71f None admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:47.762509 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a92a9974-d6bf-4d10-aca4-e1fdadc4b71f None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:47.763959 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a92a9974-d6bf-4d10-aca4-e1fdadc4b71f None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:47.789191 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 594/1173] 10.222.0.33 () {60 vars in 1141 bytes} [Sun Oct 9 00:44:47 2022] POST /identity/v3/projects => generated 361 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:44:49.916013 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e16bba7-2d1b-4a39-9ecd-b4984152dcd6 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:49.916787 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e16bba7-2d1b-4a39-9ecd-b4984152dcd6 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:49.916910 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e16bba7-2d1b-4a39-9ecd-b4984152dcd6 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:49.918125 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 580/1174] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:44:49 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:44:49.923382 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccdf3a92-352c-43ad-a9bd-ce94f5870686 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:49.923966 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccdf3a92-352c-43ad-a9bd-ce94f5870686 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:49.923966 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccdf3a92-352c-43ad-a9bd-ce94f5870686 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:49.942272 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-ccdf3a92-352c-43ad-a9bd-ce94f5870686 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:44:49.943426 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-ccdf3a92-352c-43ad-a9bd-ce94f5870686 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:44:50.006503 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 595/1175] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:44:49 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 09 00:44:50.275116 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e16bba7-2d1b-4a39-9ecd-b4984152dcd6 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:50.299719 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:50.375532 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e16bba7-2d1b-4a39-9ecd-b4984152dcd6 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:50.407504 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9a43804-8326-47d1-8c15-43b295df5df5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:50.408143 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9a43804-8326-47d1-8c15-43b295df5df5 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:50.408420 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9a43804-8326-47d1-8c15-43b295df5df5 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:50.408691 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9a43804-8326-47d1-8c15-43b295df5df5 None admin] PATH_INFO: `/v3/domains/a197d41794d649a2906072d7242baa4e` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:50.409794 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9a43804-8326-47d1-8c15-43b295df5df5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=a197d41794d649a2906072d7242baa4e)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:50.411203 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9a43804-8326-47d1-8c15-43b295df5df5 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:50.412769 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 581/1176] 10.222.0.33 () {56 vars in 1183 bytes} [Sun Oct 9 00:44:50 2022] GET /identity/v3/domains/a197d41794d649a2906072d7242baa4e => generated 267 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:44:50.419293 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccdf3a92-352c-43ad-a9bd-ce94f5870686 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:50.499158 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:50.516060 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccdf3a92-352c-43ad-a9bd-ce94f5870686 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:50.548270 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87d22065-ded3-4d45-a518-44ba5ade5c98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:50.548802 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d22065-ded3-4d45-a518-44ba5ade5c98 None admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:50.549078 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d22065-ded3-4d45-a518-44ba5ade5c98 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:50.549346 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d22065-ded3-4d45-a518-44ba5ade5c98 None admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:50.549983 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87d22065-ded3-4d45-a518-44ba5ade5c98 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:50.551408 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87d22065-ded3-4d45-a518-44ba5ade5c98 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:50.572658 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 596/1177] 10.222.0.33 () {60 vars in 1132 bytes} [Sun Oct 9 00:44:50 2022] POST /identity/v3/users => generated 311 bytes in 154 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:44:52.749731 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b4b0c77-7102-449d-98c9-64cb6103ae9c None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:52.750290 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b4b0c77-7102-449d-98c9-64cb6103ae9c None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:52.750290 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b4b0c77-7102-449d-98c9-64cb6103ae9c None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:52.751611 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 582/1178] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:44:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:44:52.757298 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcea1738-267d-4070-9695-a9994122973b None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:52.757718 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcea1738-267d-4070-9695-a9994122973b None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:52.757857 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcea1738-267d-4070-9695-a9994122973b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:52.776256 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-fcea1738-267d-4070-9695-a9994122973b None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:44:52.777429 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-fcea1738-267d-4070-9695-a9994122973b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:44:52.841537 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 597/1179] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:44:52 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 09 00:44:53.116275 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b4b0c77-7102-449d-98c9-64cb6103ae9c None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:53.204454 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:53.215203 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b4b0c77-7102-449d-98c9-64cb6103ae9c None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:53.247879 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81f95f62-e925-4669-b992-f28a1cc9a8d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:53.248492 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f95f62-e925-4669-b992-f28a1cc9a8d7 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:53.248773 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f95f62-e925-4669-b992-f28a1cc9a8d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:53.249047 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f95f62-e925-4669-b992-f28a1cc9a8d7 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:53.259242 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81f95f62-e925-4669-b992-f28a1cc9a8d7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:44:53.264039 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-81f95f62-e925-4669-b992-f28a1cc9a8d7 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 09 00:44:53.264039 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:44:53.264039 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:44:53.264039 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:44:53.264039 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:44:53.264039 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:44:53.264039 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:44:53.264039 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:44:53.264039 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:44:53.264039 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:44:53.264039 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:44:53.264039 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:44:53.264039 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:44:53.264039 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:44:53.264039 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:44:53.264039 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:44:53.265209 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:44:53.265209 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:44:53.265209 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:44:53.265209 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:44:53.265209 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:44:53.265209 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:44:53.265209 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:44:53.265209 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:44:53.265209 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:44:53.265209 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:44:53.265209 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:44:53.265209 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:44:53.265209 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:44:53.265209 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:44:53.265209 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:44:53.265209 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:44:53.266050 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:44:53.266050 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:44:53.266050 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:44:53.266050 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:44:53.266050 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:44:53.266050 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:44:53.266050 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:44:53.266050 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:44:53.266050 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:44:53.266050 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:44:53.266050 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 09 00:44:53.266050 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:44:53.266050 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 583/1180] 10.222.0.33 () {56 vars in 1096 bytes} [Sun Oct 9 00:44:53 2022] GET /identity/v3/roles/admin => generated 83 bytes in 149 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:44:53.271008 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcea1738-267d-4070-9695-a9994122973b None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:53.337572 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:53.368255 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcea1738-267d-4070-9695-a9994122973b None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:53.401094 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e2ea355-fa2c-4a59-bb9b-2a8a91e5a7ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:53.401673 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e2ea355-fa2c-4a59-bb9b-2a8a91e5a7ec None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:53.401943 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e2ea355-fa2c-4a59-bb9b-2a8a91e5a7ec None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:53.402205 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e2ea355-fa2c-4a59-bb9b-2a8a91e5a7ec None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:53.402849 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e2ea355-fa2c-4a59-bb9b-2a8a91e5a7ec None admin] RBAC: Adding query filter params (name=admin) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:44:53.403144 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e2ea355-fa2c-4a59-bb9b-2a8a91e5a7ec None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:53.404513 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e2ea355-fa2c-4a59-bb9b-2a8a91e5a7ec None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:53.412110 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 598/1181] 10.222.0.33 () {56 vars in 1099 bytes} [Sun Oct 9 00:44:53 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:44:53.418749 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81f95f62-e925-4669-b992-f28a1cc9a8d7 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:53.514013 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81f95f62-e925-4669-b992-f28a1cc9a8d7 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:53.547229 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-590fd142-e0b0-412b-947a-ad797eade99e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:53.547873 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-590fd142-e0b0-412b-947a-ad797eade99e None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:53.548154 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-590fd142-e0b0-412b-947a-ad797eade99e None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:53.548425 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-590fd142-e0b0-412b-947a-ad797eade99e None admin] PATH_INFO: `/v3/users/e7e7b613fe5f40caba88eb264569e552` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:53.562121 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-590fd142-e0b0-412b-947a-ad797eade99e None admin] RBAC: Authorizing `identity:get_user(user_id=e7e7b613fe5f40caba88eb264569e552)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:53.563542 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-590fd142-e0b0-412b-947a-ad797eade99e None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:53.565117 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 584/1182] 10.222.0.33 () {56 vars in 1177 bytes} [Sun Oct 9 00:44:53 2022] GET /identity/v3/users/e7e7b613fe5f40caba88eb264569e552 => generated 311 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:44:53.570846 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e2ea355-fa2c-4a59-bb9b-2a8a91e5a7ec None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:53.667024 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e2ea355-fa2c-4a59-bb9b-2a8a91e5a7ec None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:53.693913 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:53.699672 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e3d2755-2114-4c77-b263-84e4452e9fc7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:53.700272 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e3d2755-2114-4c77-b263-84e4452e9fc7 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:53.700551 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e3d2755-2114-4c77-b263-84e4452e9fc7 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:53.700822 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e3d2755-2114-4c77-b263-84e4452e9fc7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:53.765820 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e3d2755-2114-4c77-b263-84e4452e9fc7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:53.767374 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e3d2755-2114-4c77-b263-84e4452e9fc7 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:53.862364 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 599/1183] 10.222.0.33 () {58 vars in 1282 bytes} [Sun Oct 9 00:44:53 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 292 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 09 00:44:53.868426 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-590fd142-e0b0-412b-947a-ad797eade99e None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:53.878839 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:53.964651 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-590fd142-e0b0-412b-947a-ad797eade99e None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:53.997255 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7da1c76a-e296-432e-9673-69990f40e4a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:53.997867 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7da1c76a-e296-432e-9673-69990f40e4a9 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:53.998148 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7da1c76a-e296-432e-9673-69990f40e4a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:53.998422 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7da1c76a-e296-432e-9673-69990f40e4a9 None admin] PATH_INFO: `/v3/projects/f667566482874d6892171a481151e07c` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:53.999525 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7da1c76a-e296-432e-9673-69990f40e4a9 None admin] RBAC: Authorizing `identity:get_project(project_id=f667566482874d6892171a481151e07c)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:54.000968 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7da1c76a-e296-432e-9673-69990f40e4a9 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:54.002549 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 585/1184] 10.222.0.33 () {56 vars in 1186 bytes} [Sun Oct 9 00:44:53 2022] GET /identity/v3/projects/f667566482874d6892171a481151e07c => generated 361 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:44:54.008502 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e3d2755-2114-4c77-b263-84e4452e9fc7 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:54.038229 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:54.107514 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e3d2755-2114-4c77-b263-84e4452e9fc7 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:54.140309 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8953b72d-c026-4554-9e1d-ce88c6404780 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:54.140895 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8953b72d-c026-4554-9e1d-ce88c6404780 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:54.141170 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8953b72d-c026-4554-9e1d-ce88c6404780 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:54.141438 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8953b72d-c026-4554-9e1d-ce88c6404780 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:54.142149 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8953b72d-c026-4554-9e1d-ce88c6404780 None admin] RBAC: Adding query filter params (role.id=092036ee46ac427baf4d56468575f697, scope.project.id=f667566482874d6892171a481151e07c, user.id=e7e7b613fe5f40caba88eb264569e552) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:44:54.142453 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8953b72d-c026-4554-9e1d-ce88c6404780 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:54.143916 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8953b72d-c026-4554-9e1d-ce88c6404780 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:54.149104 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 600/1185] 10.222.0.33 () {56 vars in 1374 bytes} [Sun Oct 9 00:44:54 2022] GET /identity/v3/role_assignments?user.id=e7e7b613fe5f40caba88eb264569e552&scope.project.id=f667566482874d6892171a481151e07c&role.id=092036ee46ac427baf4d56468575f697 => generated 259 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:44:56.249307 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e603ab-02bc-443e-bc1c-b146230a78e8 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:56.249826 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e603ab-02bc-443e-bc1c-b146230a78e8 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:56.249826 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e603ab-02bc-443e-bc1c-b146230a78e8 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:56.251175 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 586/1186] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:44:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:44:56.256369 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb822d57-19f8-482c-968a-216f7e4bd1b3 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:56.256761 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb822d57-19f8-482c-968a-216f7e4bd1b3 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:56.256944 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb822d57-19f8-482c-968a-216f7e4bd1b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:56.275173 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-cb822d57-19f8-482c-968a-216f7e4bd1b3 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:44:56.276379 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-cb822d57-19f8-482c-968a-216f7e4bd1b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:44:56.340404 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 601/1187] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:44:56 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 09 00:44:56.607276 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0e603ab-02bc-443e-bc1c-b146230a78e8 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:56.681815 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:56.704586 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0e603ab-02bc-443e-bc1c-b146230a78e8 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:56.736558 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f44f41e-3f8b-4e01-8e8c-2d137f4400a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:56.737162 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f44f41e-3f8b-4e01-8e8c-2d137f4400a1 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:56.737436 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f44f41e-3f8b-4e01-8e8c-2d137f4400a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:56.737702 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f44f41e-3f8b-4e01-8e8c-2d137f4400a1 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:56.747847 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f44f41e-3f8b-4e01-8e8c-2d137f4400a1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:44:56.752509 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-0f44f41e-3f8b-4e01-8e8c-2d137f4400a1 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 09 00:44:56.752509 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:44:56.752509 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:44:56.752509 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:44:56.752509 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:44:56.752509 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:44:56.752509 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:44:56.752509 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:44:56.752509 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:44:56.752509 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:44:56.752509 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:44:56.752509 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:44:56.752509 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:44:56.752509 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:44:56.752509 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:44:56.752509 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:44:56.753584 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:44:56.753584 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:44:56.753584 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:44:56.753584 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:44:56.753584 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:44:56.753584 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:44:56.753584 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:44:56.753584 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:44:56.753584 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:44:56.753584 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:44:56.753584 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:44:56.753584 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:44:56.753584 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:44:56.753584 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:44:56.753584 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:44:56.753584 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:44:56.754703 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:44:56.754703 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:44:56.754703 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:44:56.754703 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:44:56.754703 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:44:56.754703 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:44:56.754703 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:44:56.754703 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:44:56.754703 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:44:56.754703 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:44:56.754703 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 09 00:44:56.754703 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:44:56.754703 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 587/1188] 10.222.0.33 () {56 vars in 1096 bytes} [Sun Oct 9 00:44:56 2022] GET /identity/v3/roles/admin => generated 83 bytes in 147 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:44:56.759527 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb822d57-19f8-482c-968a-216f7e4bd1b3 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:56.785191 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:56.856123 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb822d57-19f8-482c-968a-216f7e4bd1b3 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:56.889414 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72f443a9-46db-42e4-9942-46a2323bed70 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:56.889939 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72f443a9-46db-42e4-9942-46a2323bed70 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:56.890218 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72f443a9-46db-42e4-9942-46a2323bed70 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:56.890488 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72f443a9-46db-42e4-9942-46a2323bed70 None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:56.891136 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72f443a9-46db-42e4-9942-46a2323bed70 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:44:56.891433 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72f443a9-46db-42e4-9942-46a2323bed70 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:56.892815 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72f443a9-46db-42e4-9942-46a2323bed70 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:56.901131 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 602/1189] 10.222.0.33 () {56 vars in 1099 bytes} [Sun Oct 9 00:44:56 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:44:56.906744 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f44f41e-3f8b-4e01-8e8c-2d137f4400a1 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:57.001631 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f44f41e-3f8b-4e01-8e8c-2d137f4400a1 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:57.033656 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b525bee8-fe50-4626-b591-db607027b57f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:57.034260 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b525bee8-fe50-4626-b591-db607027b57f None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:57.034535 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b525bee8-fe50-4626-b591-db607027b57f None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:57.034803 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b525bee8-fe50-4626-b591-db607027b57f None admin] PATH_INFO: `/v3/users/e7e7b613fe5f40caba88eb264569e552` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:57.035916 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b525bee8-fe50-4626-b591-db607027b57f None admin] RBAC: Authorizing `identity:get_user(user_id=e7e7b613fe5f40caba88eb264569e552)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:57.037351 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b525bee8-fe50-4626-b591-db607027b57f None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:57.038867 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 588/1190] 10.222.0.33 () {56 vars in 1177 bytes} [Sun Oct 9 00:44:56 2022] GET /identity/v3/users/e7e7b613fe5f40caba88eb264569e552 => generated 311 bytes in 133 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:44:57.044766 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72f443a9-46db-42e4-9942-46a2323bed70 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:57.131150 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:57.143984 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72f443a9-46db-42e4-9942-46a2323bed70 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:57.176381 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85275916-7ce1-4f72-be77-2643621f2cbd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:57.176997 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85275916-7ce1-4f72-be77-2643621f2cbd None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:57.177279 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85275916-7ce1-4f72-be77-2643621f2cbd None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:57.177553 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85275916-7ce1-4f72-be77-2643621f2cbd None admin] PATH_INFO: `/v3/projects/f667566482874d6892171a481151e07c` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:57.178647 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85275916-7ce1-4f72-be77-2643621f2cbd None admin] RBAC: Authorizing `identity:get_project(project_id=f667566482874d6892171a481151e07c)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:57.180098 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85275916-7ce1-4f72-be77-2643621f2cbd None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:57.182246 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 603/1191] 10.222.0.33 () {56 vars in 1186 bytes} [Sun Oct 9 00:44:57 2022] GET /identity/v3/projects/f667566482874d6892171a481151e07c => generated 361 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:44:57.188182 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b525bee8-fe50-4626-b591-db607027b57f None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:57.191838 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:57.282919 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b525bee8-fe50-4626-b591-db607027b57f None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:57.315002 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c8d0942-fc18-4f93-9483-5b7ed1c786c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:57.315712 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8d0942-fc18-4f93-9483-5b7ed1c786c2 None admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:57.315992 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8d0942-fc18-4f93-9483-5b7ed1c786c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:57.316262 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8d0942-fc18-4f93-9483-5b7ed1c786c2 None admin] PATH_INFO: `/v3/projects/f667566482874d6892171a481151e07c/users/e7e7b613fe5f40caba88eb264569e552/roles/092036ee46ac427baf4d56468575f697` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:57.318094 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c8d0942-fc18-4f93-9483-5b7ed1c786c2 None admin] RBAC: Authorizing `identity:create_grant(project_id=f667566482874d6892171a481151e07c, user_id=e7e7b613fe5f40caba88eb264569e552, role_id=092036ee46ac427baf4d56468575f697)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:44:57.319490 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c8d0942-fc18-4f93-9483-5b7ed1c786c2 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:44:57.331583 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 589/1192] 10.222.0.33 () {58 vars in 1439 bytes} [Sun Oct 9 00:44:57 2022] PUT /identity/v3/projects/f667566482874d6892171a481151e07c/users/e7e7b613fe5f40caba88eb264569e552/roles/092036ee46ac427baf4d56468575f697 => generated 0 bytes in 144 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:44:59.438296 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ab51d1-ff93-4388-b60e-cd4f0be5cb02 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:59.438834 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ab51d1-ff93-4388-b60e-cd4f0be5cb02 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:59.438834 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ab51d1-ff93-4388-b60e-cd4f0be5cb02 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:59.440190 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 604/1193] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:44:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:44:59.445482 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44214486-70a5-42c8-8298-d90dabffaefd None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:59.445890 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44214486-70a5-42c8-8298-d90dabffaefd None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:59.446029 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44214486-70a5-42c8-8298-d90dabffaefd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:59.464270 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-44214486-70a5-42c8-8298-d90dabffaefd None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:44:59.465471 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-44214486-70a5-42c8-8298-d90dabffaefd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:44:59.521288 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:44:59.530835 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 590/1194] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:44:59 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 09 00:44:59.802397 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9ab51d1-ff93-4388-b60e-cd4f0be5cb02 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:44:59.901401 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9ab51d1-ff93-4388-b60e-cd4f0be5cb02 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:44:59.934614 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2d2c2da-ec44-47f3-8d11-608ccd4ee9ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:44:59.935222 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2d2c2da-ec44-47f3-8d11-608ccd4ee9ff None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:44:59.935503 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2d2c2da-ec44-47f3-8d11-608ccd4ee9ff None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:44:59.935804 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2d2c2da-ec44-47f3-8d11-608ccd4ee9ff None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:44:59.945968 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2d2c2da-ec44-47f3-8d11-608ccd4ee9ff None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:44:59.950693 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-f2d2c2da-ec44-47f3-8d11-608ccd4ee9ff None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 09 00:44:59.950693 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:44:59.950693 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:44:59.950693 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:44:59.950693 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:44:59.950693 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:44:59.950693 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:44:59.950693 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:44:59.950693 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:44:59.950693 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:44:59.950693 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:44:59.950693 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:44:59.950693 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:44:59.950693 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:44:59.950693 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:44:59.950693 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:44:59.951801 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:44:59.951801 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:44:59.951801 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:44:59.951801 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:44:59.951801 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:44:59.951801 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:44:59.951801 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:44:59.951801 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:44:59.951801 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:44:59.951801 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:44:59.951801 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:44:59.951801 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:44:59.951801 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:44:59.951801 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:44:59.951801 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:44:59.951801 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:44:59.952626 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:44:59.952626 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:44:59.952626 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:44:59.952626 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:44:59.952626 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:44:59.952626 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:44:59.952626 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:44:59.952626 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:44:59.952626 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:44:59.952626 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:44:59.952626 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 09 00:44:59.952626 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:44:59.952626 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 605/1195] 10.222.0.33 () {56 vars in 1096 bytes} [Sun Oct 9 00:44:59 2022] GET /identity/v3/roles/admin => generated 83 bytes in 150 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:44:59.957712 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44214486-70a5-42c8-8298-d90dabffaefd None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:45:00.052864 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44214486-70a5-42c8-8298-d90dabffaefd None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:45:00.084986 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37d0b6ff-bc23-48da-bf2b-0764a11d51a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:45:00.085581 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d0b6ff-bc23-48da-bf2b-0764a11d51a1 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:45:00.085859 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d0b6ff-bc23-48da-bf2b-0764a11d51a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:45:00.086129 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d0b6ff-bc23-48da-bf2b-0764a11d51a1 None admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:45:00.086780 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37d0b6ff-bc23-48da-bf2b-0764a11d51a1 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:45:00.087081 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37d0b6ff-bc23-48da-bf2b-0764a11d51a1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:45:00.089004 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37d0b6ff-bc23-48da-bf2b-0764a11d51a1 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:45:00.096807 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 591/1196] 10.222.0.33 () {56 vars in 1099 bytes} [Sun Oct 9 00:44:59 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:45:00.102575 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2d2c2da-ec44-47f3-8d11-608ccd4ee9ff None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:45:00.121758 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:45:00.201233 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2d2c2da-ec44-47f3-8d11-608ccd4ee9ff None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:45:00.233126 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88eb0fad-34cb-4396-93fa-325d39c4d092 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:45:00.233741 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88eb0fad-34cb-4396-93fa-325d39c4d092 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:45:00.234023 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88eb0fad-34cb-4396-93fa-325d39c4d092 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:45:00.234296 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88eb0fad-34cb-4396-93fa-325d39c4d092 None admin] PATH_INFO: `/v3/users/e7e7b613fe5f40caba88eb264569e552` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:45:00.235394 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88eb0fad-34cb-4396-93fa-325d39c4d092 None admin] RBAC: Authorizing `identity:get_user(user_id=e7e7b613fe5f40caba88eb264569e552)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:45:00.236854 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88eb0fad-34cb-4396-93fa-325d39c4d092 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:45:00.238383 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 606/1197] 10.222.0.33 () {56 vars in 1177 bytes} [Sun Oct 9 00:45:00 2022] GET /identity/v3/users/e7e7b613fe5f40caba88eb264569e552 => generated 311 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:45:00.244435 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37d0b6ff-bc23-48da-bf2b-0764a11d51a1 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:45:00.298088 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:45:00.340886 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37d0b6ff-bc23-48da-bf2b-0764a11d51a1 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:45:00.373567 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ff4ef38-e65a-48c3-ab0f-d05c7bb0f16a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:45:00.374157 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ff4ef38-e65a-48c3-ab0f-d05c7bb0f16a None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:45:00.374437 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ff4ef38-e65a-48c3-ab0f-d05c7bb0f16a None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:45:00.374708 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ff4ef38-e65a-48c3-ab0f-d05c7bb0f16a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:45:00.439571 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ff4ef38-e65a-48c3-ab0f-d05c7bb0f16a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:45:00.441015 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ff4ef38-e65a-48c3-ab0f-d05c7bb0f16a None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:45:00.492167 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:45:00.534893 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 592/1198] 10.222.0.33 () {58 vars in 1282 bytes} [Sun Oct 9 00:45:00 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 291 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 09 00:45:00.540977 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88eb0fad-34cb-4396-93fa-325d39c4d092 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:45:00.621172 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:45:00.637169 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88eb0fad-34cb-4396-93fa-325d39c4d092 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:45:00.670165 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2fee572-4294-44b3-8040-4b9eefcc15dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:45:00.670771 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2fee572-4294-44b3-8040-4b9eefcc15dc None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:45:00.671045 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2fee572-4294-44b3-8040-4b9eefcc15dc None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:45:00.671314 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2fee572-4294-44b3-8040-4b9eefcc15dc None admin] PATH_INFO: `/v3/projects/f667566482874d6892171a481151e07c` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:45:00.672423 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2fee572-4294-44b3-8040-4b9eefcc15dc None admin] RBAC: Authorizing `identity:get_project(project_id=f667566482874d6892171a481151e07c)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:45:00.673847 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2fee572-4294-44b3-8040-4b9eefcc15dc None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:45:00.675424 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 607/1199] 10.222.0.33 () {56 vars in 1186 bytes} [Sun Oct 9 00:45:00 2022] GET /identity/v3/projects/f667566482874d6892171a481151e07c => generated 361 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:45:00.681502 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ff4ef38-e65a-48c3-ab0f-d05c7bb0f16a None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:45:00.775606 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ff4ef38-e65a-48c3-ab0f-d05c7bb0f16a None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:45:00.808061 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc61685e-0c1c-479d-b73d-8440f92fb07f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:45:00.808642 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc61685e-0c1c-479d-b73d-8440f92fb07f None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:45:00.808921 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc61685e-0c1c-479d-b73d-8440f92fb07f None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:45:00.809192 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc61685e-0c1c-479d-b73d-8440f92fb07f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:45:00.809899 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc61685e-0c1c-479d-b73d-8440f92fb07f None admin] RBAC: Adding query filter params (role.id=092036ee46ac427baf4d56468575f697, scope.project.id=f667566482874d6892171a481151e07c, user.id=e7e7b613fe5f40caba88eb264569e552) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:45:00.810203 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc61685e-0c1c-479d-b73d-8440f92fb07f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:45:00.811625 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc61685e-0c1c-479d-b73d-8440f92fb07f None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:45:00.817066 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 593/1200] 10.222.0.33 () {56 vars in 1374 bytes} [Sun Oct 9 00:45:00 2022] GET /identity/v3/role_assignments?user.id=e7e7b613fe5f40caba88eb264569e552&scope.project.id=f667566482874d6892171a481151e07c&role.id=092036ee46ac427baf4d56468575f697 => generated 608 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:45:10.665921 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=24 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:45:10.814590 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=90 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:45:56.352867 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b3d9af-6cc5-4391-b78b-f5c09aeade85 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:45:56.353515 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b3d9af-6cc5-4391-b78b-f5c09aeade85 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:45:56.353515 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b3d9af-6cc5-4391-b78b-f5c09aeade85 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:45:56.354775 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 608/1201] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:45:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:45:56.360934 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c51bfab9-c189-42e0-9dbf-884ed758127d None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:45:56.361414 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c51bfab9-c189-42e0-9dbf-884ed758127d None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:45:56.361498 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c51bfab9-c189-42e0-9dbf-884ed758127d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:45:56.380744 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-c51bfab9-c189-42e0-9dbf-884ed758127d None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:45:56.381993 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-c51bfab9-c189-42e0-9dbf-884ed758127d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:45:56.479642 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 594/1202] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:45:56 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 09 00:45:56.754107 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b3d9af-6cc5-4391-b78b-f5c09aeade85 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:45:56.907301 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b3d9af-6cc5-4391-b78b-f5c09aeade85 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:45:56.943477 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76f2424e-d3e8-41e0-a7bf-3ae566c3f66c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:45:56.944146 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76f2424e-d3e8-41e0-a7bf-3ae566c3f66c None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:45:56.944430 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76f2424e-d3e8-41e0-a7bf-3ae566c3f66c None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:45:56.944707 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76f2424e-d3e8-41e0-a7bf-3ae566c3f66c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:45:56.959352 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76f2424e-d3e8-41e0-a7bf-3ae566c3f66c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:45:56.961017 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76f2424e-d3e8-41e0-a7bf-3ae566c3f66c None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:45:56.976885 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-76f2424e-d3e8-41e0-a7bf-3ae566c3f66c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:45:56.976885 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:45:56.976885 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:45:56.976885 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:45:56.976885 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:45:56.976885 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:45:56.976885 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:45:56.976885 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:45:56.976885 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:45:56.976885 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:45:56.976885 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:45:56.976885 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:45:56.976885 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:45:56.976885 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:45:56.976885 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:45:56.976885 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:45:56.978140 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:45:56.978140 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:45:56.978140 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:45:56.978140 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:45:56.978140 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:45:56.978140 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:45:56.978140 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:45:56.978140 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:45:56.978140 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:45:56.978140 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:45:56.978140 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:45:56.978140 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:45:56.978140 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:45:56.978140 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:45:56.978140 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:45:56.978140 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:45:56.978140 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:45:56.979176 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:45:56.979176 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:45:56.979176 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:45:56.979176 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 609/1203] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:45:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 224 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:45:56.984250 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c51bfab9-c189-42e0-9dbf-884ed758127d None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:45:57.119918 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c51bfab9-c189-42e0-9dbf-884ed758127d None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:45:57.160146 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:45:57.167522 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45cb1db6-2cbc-4e2e-b4bc-01ddb8b263c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:45:57.168275 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45cb1db6-2cbc-4e2e-b4bc-01ddb8b263c4 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:45:57.168663 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45cb1db6-2cbc-4e2e-b4bc-01ddb8b263c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:45:57.169027 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45cb1db6-2cbc-4e2e-b4bc-01ddb8b263c4 None admin] PATH_INFO: `/v3/domains` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:45:57.169779 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45cb1db6-2cbc-4e2e-b4bc-01ddb8b263c4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:45:57.170485 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45cb1db6-2cbc-4e2e-b4bc-01ddb8b263c4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:45:57.172110 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45cb1db6-2cbc-4e2e-b4bc-01ddb8b263c4 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:45:57.190625 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76f2424e-d3e8-41e0-a7bf-3ae566c3f66c None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:45:57.194475 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 595/1204] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:45:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:45:57.265389 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:45:57.327957 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76f2424e-d3e8-41e0-a7bf-3ae566c3f66c None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:45:57.372541 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad24e25d-3a02-4f7f-b2a4-fba682f37b65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:45:57.373214 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad24e25d-3a02-4f7f-b2a4-fba682f37b65 None admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:45:57.373841 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad24e25d-3a02-4f7f-b2a4-fba682f37b65 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:45:57.373841 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad24e25d-3a02-4f7f-b2a4-fba682f37b65 None admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:45:57.374374 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad24e25d-3a02-4f7f-b2a4-fba682f37b65 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:45:57.375856 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad24e25d-3a02-4f7f-b2a4-fba682f37b65 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:45:57.404526 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 610/1205] 10.222.0.33 () {60 vars in 1132 bytes} [Sun Oct 9 00:45:57 2022] POST /identity/v3/users => generated 251 bytes in 214 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:45:59.863196 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a20aa7-8e54-4dc7-8bf1-a22dbcf4d82d None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:45:59.863817 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a20aa7-8e54-4dc7-8bf1-a22dbcf4d82d None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:45:59.863817 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a20aa7-8e54-4dc7-8bf1-a22dbcf4d82d None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:45:59.865378 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 596/1206] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:45:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 09 00:45:59.872675 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd11f112-ef65-4811-bc8b-956d3bcf97ef None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:45:59.873147 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd11f112-ef65-4811-bc8b-956d3bcf97ef None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:45:59.873586 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd11f112-ef65-4811-bc8b-956d3bcf97ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:45:59.896362 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-dd11f112-ef65-4811-bc8b-956d3bcf97ef None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:45:59.897848 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-dd11f112-ef65-4811-bc8b-956d3bcf97ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:45:59.995352 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 611/1207] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:45:59 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 09 00:46:00.299483 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7a20aa7-8e54-4dc7-8bf1-a22dbcf4d82d None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:00.429042 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7a20aa7-8e54-4dc7-8bf1-a22dbcf4d82d None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:00.469262 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0701901c-1601-439b-9695-04dfdb0be485 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:00.469942 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0701901c-1601-439b-9695-04dfdb0be485 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:00.470344 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0701901c-1601-439b-9695-04dfdb0be485 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:00.470460 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0701901c-1601-439b-9695-04dfdb0be485 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:00.484105 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0701901c-1601-439b-9695-04dfdb0be485 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:46:00.489138 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-0701901c-1601-439b-9695-04dfdb0be485 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 09 00:46:00.489138 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:46:00.489138 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:46:00.489138 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:46:00.489138 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:46:00.489138 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:46:00.489138 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:46:00.489138 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:46:00.489138 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:46:00.489138 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:46:00.489138 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:46:00.489138 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:46:00.489138 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:46:00.489138 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:46:00.489138 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:46:00.489138 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:46:00.490294 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:46:00.490294 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:46:00.490294 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:46:00.490294 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:46:00.490294 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:46:00.490294 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:46:00.490294 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:46:00.490294 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:46:00.490294 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:46:00.490294 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:46:00.490294 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:46:00.490294 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:46:00.490294 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:46:00.490294 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:46:00.490294 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:46:00.490294 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:46:00.491407 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:46:00.491407 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:46:00.491407 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:46:00.491407 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:46:00.491407 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:46:00.491407 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:46:00.491407 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:46:00.491407 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:46:00.491407 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:46:00.491407 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:46:00.491407 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 09 00:46:00.491407 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:46:00.495532 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 597/1208] 10.222.0.33 () {56 vars in 1102 bytes} [Sun Oct 9 00:46:00 2022] GET /identity/v3/roles/service => generated 85 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:46:00.497041 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd11f112-ef65-4811-bc8b-956d3bcf97ef None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:00.550297 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:46:00.628778 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd11f112-ef65-4811-bc8b-956d3bcf97ef None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:00.671687 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-738e8a78-e21b-44de-b594-7b3f75a41223 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:00.672654 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738e8a78-e21b-44de-b594-7b3f75a41223 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:00.672972 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738e8a78-e21b-44de-b594-7b3f75a41223 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:00.673371 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738e8a78-e21b-44de-b594-7b3f75a41223 None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:00.674097 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-738e8a78-e21b-44de-b594-7b3f75a41223 None admin] RBAC: Adding query filter params (name=service) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:46:00.674588 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-738e8a78-e21b-44de-b594-7b3f75a41223 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:46:00.676297 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-738e8a78-e21b-44de-b594-7b3f75a41223 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:46:00.689161 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 612/1209] 10.222.0.33 () {56 vars in 1103 bytes} [Sun Oct 9 00:46:00 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:46:00.695462 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0701901c-1601-439b-9695-04dfdb0be485 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:00.756925 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:46:00.825473 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0701901c-1601-439b-9695-04dfdb0be485 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:00.867091 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c561c61c-a3f0-4f40-a6c9-90ab72220cb8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:00.868537 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c561c61c-a3f0-4f40-a6c9-90ab72220cb8 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:00.868984 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c561c61c-a3f0-4f40-a6c9-90ab72220cb8 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:00.869402 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c561c61c-a3f0-4f40-a6c9-90ab72220cb8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:00.881330 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c561c61c-a3f0-4f40-a6c9-90ab72220cb8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:46:00.882971 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c561c61c-a3f0-4f40-a6c9-90ab72220cb8 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:46:00.898319 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-c561c61c-a3f0-4f40-a6c9-90ab72220cb8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:46:00.898319 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:46:00.898319 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:46:00.898319 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:46:00.898319 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:46:00.898319 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:46:00.898319 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:46:00.898319 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:46:00.898319 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:46:00.898319 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:46:00.898319 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:46:00.898319 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:46:00.898319 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:46:00.898319 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:46:00.898319 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:46:00.898319 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:46:00.899326 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:46:00.899326 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:46:00.899326 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:46:00.899326 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:46:00.899326 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:46:00.899326 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:46:00.899326 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:46:00.899326 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:46:00.899326 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:46:00.899326 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:46:00.899326 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:46:00.899326 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:46:00.899326 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:46:00.899326 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:46:00.899326 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:46:00.899326 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:46:00.899326 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:46:00.905903 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:46:00.905903 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:46:00.905903 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:46:00.905903 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 598/1210] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:46:00 2022] GET /identity/v3/domains/Default => generated 87 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:46:00.906186 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-738e8a78-e21b-44de-b594-7b3f75a41223 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:01.011298 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:46:01.014515 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-738e8a78-e21b-44de-b594-7b3f75a41223 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:01.055629 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54855c22-6def-445c-88eb-dcb13d91e2a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:01.056251 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54855c22-6def-445c-88eb-dcb13d91e2a9 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:01.056550 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54855c22-6def-445c-88eb-dcb13d91e2a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:01.056822 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54855c22-6def-445c-88eb-dcb13d91e2a9 None admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:01.057463 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54855c22-6def-445c-88eb-dcb13d91e2a9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:46:01.057767 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54855c22-6def-445c-88eb-dcb13d91e2a9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:46:01.059263 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54855c22-6def-445c-88eb-dcb13d91e2a9 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:46:01.078446 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 613/1211] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:46:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:46:01.082272 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c561c61c-a3f0-4f40-a6c9-90ab72220cb8 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:01.193114 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c561c61c-a3f0-4f40-a6c9-90ab72220cb8 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:01.194946 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:46:01.229825 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23f21c3a-7ba7-41fc-82bb-dcdc8659bf46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:01.230557 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23f21c3a-7ba7-41fc-82bb-dcdc8659bf46 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:01.230796 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23f21c3a-7ba7-41fc-82bb-dcdc8659bf46 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:01.231042 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23f21c3a-7ba7-41fc-82bb-dcdc8659bf46 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:01.238982 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23f21c3a-7ba7-41fc-82bb-dcdc8659bf46 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:46:01.240485 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23f21c3a-7ba7-41fc-82bb-dcdc8659bf46 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:46:01.252046 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-23f21c3a-7ba7-41fc-82bb-dcdc8659bf46 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 09 00:46:01.252046 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:46:01.252046 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:46:01.252046 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:46:01.252046 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:46:01.252046 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:46:01.252046 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:46:01.252046 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:46:01.252046 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:46:01.252046 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:46:01.252046 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:46:01.252046 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:46:01.252046 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 09 00:46:01.252046 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 09 00:46:01.252046 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 09 00:46:01.252046 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 09 00:46:01.253105 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:46:01.253105 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:46:01.253105 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 09 00:46:01.253105 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 09 00:46:01.253105 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 09 00:46:01.253105 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 09 00:46:01.253105 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:46:01.253105 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:46:01.253105 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:46:01.253105 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:46:01.253105 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:46:01.253105 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:46:01.253105 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:46:01.253105 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:46:01.253105 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:46:01.253105 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:46:01.253105 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:46:01.253105 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:46:01.254817 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:46:01.254817 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:46:01.254817 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 09 00:46:01.254817 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 09 00:46:01.254817 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 09 00:46:01.254817 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 09 00:46:01.254817 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 09 00:46:01.254817 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 09 00:46:01.254817 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 09 00:46:01.254817 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:46:01.254817 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 599/1212] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:46:01 2022] GET /identity/v3/users/placement => generated 87 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:46:01.260717 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54855c22-6def-445c-88eb-dcb13d91e2a9 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:01.375312 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54855c22-6def-445c-88eb-dcb13d91e2a9 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:01.421207 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-210f4f4a-6446-456c-90e0-49b9bb70655e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:01.421834 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-210f4f4a-6446-456c-90e0-49b9bb70655e None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:01.422185 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-210f4f4a-6446-456c-90e0-49b9bb70655e None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:01.422385 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-210f4f4a-6446-456c-90e0-49b9bb70655e None admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:01.423128 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-210f4f4a-6446-456c-90e0-49b9bb70655e None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:46:01.423415 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-210f4f4a-6446-456c-90e0-49b9bb70655e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:46:01.424887 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-210f4f4a-6446-456c-90e0-49b9bb70655e None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:46:01.443213 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 614/1213] 10.222.0.33 () {56 vars in 1143 bytes} [Sun Oct 9 00:46:01 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:46:01.449147 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23f21c3a-7ba7-41fc-82bb-dcdc8659bf46 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:01.565514 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23f21c3a-7ba7-41fc-82bb-dcdc8659bf46 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:01.606576 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1cbb9c0-a517-44f3-a771-6fdf2ee55ba3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:01.607176 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1cbb9c0-a517-44f3-a771-6fdf2ee55ba3 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:01.607451 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1cbb9c0-a517-44f3-a771-6fdf2ee55ba3 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:01.607758 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1cbb9c0-a517-44f3-a771-6fdf2ee55ba3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:01.642307 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:46:01.694444 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1cbb9c0-a517-44f3-a771-6fdf2ee55ba3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:46:01.696353 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1cbb9c0-a517-44f3-a771-6fdf2ee55ba3 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:46:01.827728 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 600/1214] 10.222.0.33 () {58 vars in 1282 bytes} [Sun Oct 9 00:46:01 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 379 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 09 00:46:01.835104 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-210f4f4a-6446-456c-90e0-49b9bb70655e None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:01.854590 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:46:01.963036 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-210f4f4a-6446-456c-90e0-49b9bb70655e None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:02.009092 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cf91fcf-1133-49e8-b5c9-e83055dcef15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:02.009884 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf91fcf-1133-49e8-b5c9-e83055dcef15 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:02.010223 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf91fcf-1133-49e8-b5c9-e83055dcef15 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:02.010554 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cf91fcf-1133-49e8-b5c9-e83055dcef15 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:02.024745 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cf91fcf-1133-49e8-b5c9-e83055dcef15 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:46:02.026567 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cf91fcf-1133-49e8-b5c9-e83055dcef15 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:46:02.046253 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-7cf91fcf-1133-49e8-b5c9-e83055dcef15 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:46:02.046253 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:46:02.046253 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:46:02.046253 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:46:02.046253 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:46:02.046253 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:46:02.046253 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:46:02.046253 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:46:02.046253 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:46:02.046253 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:46:02.046253 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:46:02.046253 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:46:02.046253 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:46:02.046253 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:46:02.046253 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:46:02.046253 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:46:02.047538 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:46:02.047538 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:46:02.047538 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:46:02.047538 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:46:02.047538 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:46:02.047538 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:46:02.047538 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:46:02.047538 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:46:02.047538 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:46:02.047538 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:46:02.047538 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:46:02.047538 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:46:02.047538 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:46:02.047538 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:46:02.047538 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:46:02.047538 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:46:02.047538 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:46:02.048589 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:46:02.048589 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:46:02.048589 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:46:02.048589 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 615/1215] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:46:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:46:02.054728 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1cbb9c0-a517-44f3-a771-6fdf2ee55ba3 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:02.131511 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:46:02.183528 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1cbb9c0-a517-44f3-a771-6fdf2ee55ba3 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:02.226826 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-804f24ee-25db-400c-89b3-abe5110a593f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:02.228187 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-804f24ee-25db-400c-89b3-abe5110a593f None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:02.228473 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-804f24ee-25db-400c-89b3-abe5110a593f None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:02.228752 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-804f24ee-25db-400c-89b3-abe5110a593f None admin] PATH_INFO: `/v3/domains` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:02.229414 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-804f24ee-25db-400c-89b3-abe5110a593f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:46:02.229757 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-804f24ee-25db-400c-89b3-abe5110a593f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:46:02.231094 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-804f24ee-25db-400c-89b3-abe5110a593f None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:46:02.245737 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 601/1216] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:46:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:46:02.252572 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cf91fcf-1133-49e8-b5c9-e83055dcef15 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:02.333451 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:46:02.368777 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cf91fcf-1133-49e8-b5c9-e83055dcef15 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:02.405236 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5238525e-07dd-43f5-af47-c00db27b063b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:02.405914 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5238525e-07dd-43f5-af47-c00db27b063b None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:02.406146 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5238525e-07dd-43f5-af47-c00db27b063b None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:02.406416 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5238525e-07dd-43f5-af47-c00db27b063b None admin] PATH_INFO: `/v3/projects/service` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:02.413390 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5238525e-07dd-43f5-af47-c00db27b063b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:46:02.414831 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5238525e-07dd-43f5-af47-c00db27b063b None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:46:02.426216 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-5238525e-07dd-43f5-af47-c00db27b063b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 09 00:46:02.426216 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:46:02.426216 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:46:02.426216 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:46:02.426216 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:46:02.426216 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:46:02.426216 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:46:02.426216 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:46:02.426216 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:46:02.426216 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:46:02.426216 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:46:02.426216 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:46:02.426216 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 09 00:46:02.426216 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 09 00:46:02.426216 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 09 00:46:02.427330 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 09 00:46:02.427330 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:46:02.427330 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:46:02.427330 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:46:02.427330 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:46:02.427330 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:46:02.427330 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:46:02.427330 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:46:02.427330 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:46:02.427330 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:46:02.427330 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:46:02.427330 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:46:02.427330 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:46:02.427330 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:46:02.427330 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:46:02.427330 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:46:02.427330 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:46:02.429005 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 09 00:46:02.429005 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 09 00:46:02.429005 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 09 00:46:02.429005 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 09 00:46:02.429005 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 09 00:46:02.429005 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 09 00:46:02.429005 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 09 00:46:02.429005 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:46:02.429005 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 616/1217] 10.222.0.33 () {56 vars in 1111 bytes} [Sun Oct 9 00:46:02 2022] GET /identity/v3/projects/service => generated 88 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:46:02.433268 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-804f24ee-25db-400c-89b3-abe5110a593f None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:02.549196 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-804f24ee-25db-400c-89b3-abe5110a593f None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:02.571335 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:46:02.584200 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1d1009d-e26e-4065-990e-9da3eb77e7c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:02.584826 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1d1009d-e26e-4065-990e-9da3eb77e7c4 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:02.585080 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1d1009d-e26e-4065-990e-9da3eb77e7c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:02.585376 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1d1009d-e26e-4065-990e-9da3eb77e7c4 None admin] PATH_INFO: `/v3/projects` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:02.586078 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1d1009d-e26e-4065-990e-9da3eb77e7c4 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:46:02.586478 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1d1009d-e26e-4065-990e-9da3eb77e7c4 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:46:02.588173 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1d1009d-e26e-4065-990e-9da3eb77e7c4 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:46:02.602962 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 602/1218] 10.222.0.33 () {56 vars in 1148 bytes} [Sun Oct 9 00:46:02 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:46:02.610595 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5238525e-07dd-43f5-af47-c00db27b063b None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:02.727278 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5238525e-07dd-43f5-af47-c00db27b063b None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:02.759321 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:46:02.767406 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0906c79b-1360-4853-9fd2-a6c5aba0131c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:02.768189 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0906c79b-1360-4853-9fd2-a6c5aba0131c None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:02.768467 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0906c79b-1360-4853-9fd2-a6c5aba0131c None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:02.768777 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0906c79b-1360-4853-9fd2-a6c5aba0131c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:02.769997 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0906c79b-1360-4853-9fd2-a6c5aba0131c None admin] RBAC: Adding query filter params (role.id=228232e130c5412a80d774978175d28b, scope.project.id=013c83476b1340fbb051061833c4a9c6, user.id=d25e240cb93643ed8122ccac9586936f) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:46:02.770425 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0906c79b-1360-4853-9fd2-a6c5aba0131c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:46:02.772224 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0906c79b-1360-4853-9fd2-a6c5aba0131c None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:46:02.780613 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 617/1219] 10.222.0.33 () {56 vars in 1374 bytes} [Sun Oct 9 00:46:02 2022] GET /identity/v3/role_assignments?user.id=d25e240cb93643ed8122ccac9586936f&scope.project.id=013c83476b1340fbb051061833c4a9c6&role.id=228232e130c5412a80d774978175d28b => generated 259 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:46:05.052641 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6276d23f-b03d-41f1-b5b3-37240ac093ae None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:05.053273 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6276d23f-b03d-41f1-b5b3-37240ac093ae None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:05.053273 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6276d23f-b03d-41f1-b5b3-37240ac093ae None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:05.054988 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 603/1220] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:46:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:46:05.061373 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c7908f-50bd-44d2-aff6-1b584987d910 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:05.061856 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c7908f-50bd-44d2-aff6-1b584987d910 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:05.061966 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c7908f-50bd-44d2-aff6-1b584987d910 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:05.081387 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-f6c7908f-50bd-44d2-aff6-1b584987d910 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:46:05.082561 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-f6c7908f-50bd-44d2-aff6-1b584987d910 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:46:05.155662 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 618/1221] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:46:05 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 09 00:46:05.436022 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6276d23f-b03d-41f1-b5b3-37240ac093ae None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:05.542372 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6276d23f-b03d-41f1-b5b3-37240ac093ae None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:05.580120 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74b163ff-33d6-4187-8d73-e84e026a85a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:05.580872 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74b163ff-33d6-4187-8d73-e84e026a85a2 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:05.581187 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74b163ff-33d6-4187-8d73-e84e026a85a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:05.581480 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74b163ff-33d6-4187-8d73-e84e026a85a2 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:05.592512 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74b163ff-33d6-4187-8d73-e84e026a85a2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:46:05.597933 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-74b163ff-33d6-4187-8d73-e84e026a85a2 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 09 00:46:05.597933 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:46:05.597933 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:46:05.597933 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:46:05.597933 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:46:05.597933 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:46:05.597933 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:46:05.597933 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:46:05.597933 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:46:05.597933 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:46:05.597933 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:46:05.597933 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:46:05.597933 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:46:05.597933 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:46:05.597933 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:46:05.597933 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:46:05.598945 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:46:05.598945 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:46:05.598945 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:46:05.598945 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:46:05.598945 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:46:05.598945 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:46:05.598945 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:46:05.598945 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:46:05.598945 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:46:05.598945 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:46:05.598945 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:46:05.598945 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:46:05.598945 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:46:05.598945 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:46:05.598945 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:46:05.598945 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:46:05.599784 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:46:05.599784 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:46:05.599784 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:46:05.599784 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:46:05.599784 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:46:05.599784 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:46:05.599784 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:46:05.599784 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:46:05.599784 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:46:05.599784 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:46:05.599784 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 09 00:46:05.599784 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:46:05.599784 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 604/1222] 10.222.0.33 () {56 vars in 1102 bytes} [Sun Oct 9 00:46:05 2022] GET /identity/v3/roles/service => generated 85 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:46:05.605468 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6c7908f-50bd-44d2-aff6-1b584987d910 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:05.710207 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6c7908f-50bd-44d2-aff6-1b584987d910 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:05.736706 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:46:05.748582 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-871d1841-ec3d-4738-a5f8-f19a71a67e68 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:05.749228 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-871d1841-ec3d-4738-a5f8-f19a71a67e68 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:05.749531 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-871d1841-ec3d-4738-a5f8-f19a71a67e68 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:05.749832 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-871d1841-ec3d-4738-a5f8-f19a71a67e68 None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:05.750553 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-871d1841-ec3d-4738-a5f8-f19a71a67e68 None admin] RBAC: Adding query filter params (name=service) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:46:05.750854 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-871d1841-ec3d-4738-a5f8-f19a71a67e68 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:46:05.752391 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-871d1841-ec3d-4738-a5f8-f19a71a67e68 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:46:05.761495 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 619/1223] 10.222.0.33 () {56 vars in 1103 bytes} [Sun Oct 9 00:46:05 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:46:05.768717 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74b163ff-33d6-4187-8d73-e84e026a85a2 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:05.822925 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:46:05.883042 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74b163ff-33d6-4187-8d73-e84e026a85a2 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:05.922471 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bf12d82-42ec-4c60-9499-f5eabb80d515 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:05.923099 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf12d82-42ec-4c60-9499-f5eabb80d515 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:05.923391 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf12d82-42ec-4c60-9499-f5eabb80d515 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:05.923705 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf12d82-42ec-4c60-9499-f5eabb80d515 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:05.933664 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bf12d82-42ec-4c60-9499-f5eabb80d515 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:46:05.935141 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bf12d82-42ec-4c60-9499-f5eabb80d515 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:46:05.951043 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-2bf12d82-42ec-4c60-9499-f5eabb80d515 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:46:05.951043 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:46:05.951043 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:46:05.951043 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:46:05.951043 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:46:05.951043 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:46:05.951043 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:46:05.951043 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:46:05.951043 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:46:05.951043 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:46:05.951043 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:46:05.951043 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:46:05.951043 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:46:05.951043 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:46:05.951043 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:46:05.951043 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:46:05.952130 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:46:05.952130 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:46:05.952130 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:46:05.952130 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:46:05.952130 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:46:05.952130 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:46:05.952130 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:46:05.952130 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:46:05.952130 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:46:05.952130 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:46:05.952130 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:46:05.952130 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:46:05.952130 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:46:05.952130 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:46:05.952130 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:46:05.952130 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:46:05.952130 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:46:05.953008 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:46:05.953008 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:46:05.953008 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:46:05.953008 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 605/1224] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:46:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:46:05.958104 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-871d1841-ec3d-4738-a5f8-f19a71a67e68 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:06.067873 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-871d1841-ec3d-4738-a5f8-f19a71a67e68 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:06.103840 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4647a491-51a2-4c3c-936d-b221f96c4714 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:06.104448 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4647a491-51a2-4c3c-936d-b221f96c4714 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:06.104726 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4647a491-51a2-4c3c-936d-b221f96c4714 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:06.104994 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4647a491-51a2-4c3c-936d-b221f96c4714 None admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:06.105662 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4647a491-51a2-4c3c-936d-b221f96c4714 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:46:06.105964 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4647a491-51a2-4c3c-936d-b221f96c4714 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:46:06.107315 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4647a491-51a2-4c3c-936d-b221f96c4714 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:46:06.120009 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 620/1225] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:46:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:46:06.126623 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bf12d82-42ec-4c60-9499-f5eabb80d515 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:06.225899 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:46:06.233360 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bf12d82-42ec-4c60-9499-f5eabb80d515 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:06.271298 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7297062-c8d7-4357-a3fe-c1275ab86b8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:06.272108 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7297062-c8d7-4357-a3fe-c1275ab86b8d None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:06.272461 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7297062-c8d7-4357-a3fe-c1275ab86b8d None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:06.272733 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7297062-c8d7-4357-a3fe-c1275ab86b8d None admin] PATH_INFO: `/v3/users/placement` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:06.280762 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7297062-c8d7-4357-a3fe-c1275ab86b8d None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:46:06.282203 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7297062-c8d7-4357-a3fe-c1275ab86b8d None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:46:06.293334 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-b7297062-c8d7-4357-a3fe-c1275ab86b8d None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 09 00:46:06.293334 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:46:06.293334 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:46:06.293334 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:46:06.293334 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:46:06.293334 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:46:06.293334 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:46:06.293334 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:46:06.293334 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:46:06.293334 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:46:06.293334 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:46:06.293334 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:46:06.293334 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 09 00:46:06.293334 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 09 00:46:06.293334 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 09 00:46:06.293334 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 09 00:46:06.294429 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:46:06.294429 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:46:06.294429 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 09 00:46:06.294429 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 09 00:46:06.294429 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 09 00:46:06.294429 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 09 00:46:06.294429 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:46:06.294429 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:46:06.294429 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:46:06.294429 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:46:06.294429 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:46:06.294429 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:46:06.294429 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:46:06.294429 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:46:06.294429 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:46:06.294429 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:46:06.294429 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:46:06.294429 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:46:06.295368 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:46:06.295368 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:46:06.295368 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 09 00:46:06.295368 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 09 00:46:06.295368 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 09 00:46:06.295368 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 09 00:46:06.295368 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 09 00:46:06.295368 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 09 00:46:06.295368 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 09 00:46:06.295368 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:46:06.295368 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 606/1226] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:46:06 2022] GET /identity/v3/users/placement => generated 87 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:46:06.301421 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4647a491-51a2-4c3c-936d-b221f96c4714 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:06.352016 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:46:06.407749 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4647a491-51a2-4c3c-936d-b221f96c4714 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:06.446216 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f3286c9-5f3c-4043-80b9-fda94692319e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:06.446930 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3286c9-5f3c-4043-80b9-fda94692319e None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:06.447247 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3286c9-5f3c-4043-80b9-fda94692319e None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:06.447608 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3286c9-5f3c-4043-80b9-fda94692319e None admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:06.448421 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f3286c9-5f3c-4043-80b9-fda94692319e None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:46:06.448762 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f3286c9-5f3c-4043-80b9-fda94692319e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:46:06.450428 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f3286c9-5f3c-4043-80b9-fda94692319e None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:46:06.474205 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 621/1227] 10.222.0.33 () {56 vars in 1143 bytes} [Sun Oct 9 00:46:06 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:46:06.475421 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7297062-c8d7-4357-a3fe-c1275ab86b8d None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:06.584221 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7297062-c8d7-4357-a3fe-c1275ab86b8d None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:06.619836 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-484004df-0f25-4d11-b5c0-29303a2c6015 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:06.620459 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-484004df-0f25-4d11-b5c0-29303a2c6015 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:06.620782 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-484004df-0f25-4d11-b5c0-29303a2c6015 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:06.621135 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-484004df-0f25-4d11-b5c0-29303a2c6015 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:06.633899 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-484004df-0f25-4d11-b5c0-29303a2c6015 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:46:06.635750 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-484004df-0f25-4d11-b5c0-29303a2c6015 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:46:06.651242 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-484004df-0f25-4d11-b5c0-29303a2c6015 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:46:06.651242 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:46:06.651242 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:46:06.651242 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:46:06.651242 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:46:06.651242 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:46:06.651242 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:46:06.651242 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:46:06.651242 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:46:06.651242 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:46:06.651242 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:46:06.651242 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:46:06.651242 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:46:06.651242 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:46:06.651242 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:46:06.651242 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:46:06.652254 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:46:06.652254 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:46:06.652254 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:46:06.652254 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:46:06.652254 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:46:06.652254 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:46:06.652254 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:46:06.652254 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:46:06.652254 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:46:06.652254 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:46:06.652254 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:46:06.652254 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:46:06.652254 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:46:06.652254 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:46:06.652254 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:46:06.652254 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:46:06.652254 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:46:06.653157 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:46:06.653157 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:46:06.653157 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:46:06.653157 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 607/1228] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:46:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:46:06.658349 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f3286c9-5f3c-4043-80b9-fda94692319e None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:06.768177 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f3286c9-5f3c-4043-80b9-fda94692319e None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:06.770929 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:46:06.809203 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-319da0c1-2b41-48f7-a861-db37f901e956 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:06.809840 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-319da0c1-2b41-48f7-a861-db37f901e956 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:06.810118 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-319da0c1-2b41-48f7-a861-db37f901e956 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:06.810390 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-319da0c1-2b41-48f7-a861-db37f901e956 None admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:06.811109 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-319da0c1-2b41-48f7-a861-db37f901e956 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:46:06.811391 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-319da0c1-2b41-48f7-a861-db37f901e956 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:46:06.812803 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-319da0c1-2b41-48f7-a861-db37f901e956 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:46:06.826785 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 622/1229] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:46:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:46:06.832594 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-484004df-0f25-4d11-b5c0-29303a2c6015 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:06.842836 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:46:06.940993 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-484004df-0f25-4d11-b5c0-29303a2c6015 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:06.978557 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a4845bc-1b0a-4815-bfbc-cd1edbd33a17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:06.979198 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a4845bc-1b0a-4815-bfbc-cd1edbd33a17 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:06.979474 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a4845bc-1b0a-4815-bfbc-cd1edbd33a17 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:06.979774 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a4845bc-1b0a-4815-bfbc-cd1edbd33a17 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:06.987645 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a4845bc-1b0a-4815-bfbc-cd1edbd33a17 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:46:06.989208 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a4845bc-1b0a-4815-bfbc-cd1edbd33a17 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:46:07.002195 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-0a4845bc-1b0a-4815-bfbc-cd1edbd33a17 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 09 00:46:07.002195 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:46:07.002195 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:46:07.002195 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:46:07.002195 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:46:07.002195 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:46:07.002195 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:46:07.002195 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:46:07.002195 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:46:07.002195 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:46:07.002195 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:46:07.002195 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:46:07.002195 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 09 00:46:07.002195 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 09 00:46:07.002195 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 09 00:46:07.003281 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 09 00:46:07.003281 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:46:07.003281 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:46:07.003281 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:46:07.003281 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:46:07.003281 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:46:07.003281 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:46:07.003281 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:46:07.003281 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:46:07.003281 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:46:07.003281 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:46:07.003281 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:46:07.003281 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:46:07.003281 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:46:07.003281 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:46:07.003281 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:46:07.003281 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:46:07.005041 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 09 00:46:07.005041 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 09 00:46:07.005041 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 09 00:46:07.005041 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 09 00:46:07.005041 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 09 00:46:07.005041 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 09 00:46:07.005041 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 09 00:46:07.005041 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:46:07.005041 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 608/1230] 10.222.0.33 () {56 vars in 1111 bytes} [Sun Oct 9 00:46:06 2022] GET /identity/v3/projects/service => generated 88 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:46:07.009413 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-319da0c1-2b41-48f7-a861-db37f901e956 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:07.116051 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-319da0c1-2b41-48f7-a861-db37f901e956 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:07.154208 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-152e58c1-9678-4d2d-83e1-3f17daf16e7c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:07.154875 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-152e58c1-9678-4d2d-83e1-3f17daf16e7c None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:07.155202 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-152e58c1-9678-4d2d-83e1-3f17daf16e7c None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:07.155475 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-152e58c1-9678-4d2d-83e1-3f17daf16e7c None admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:07.156179 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-152e58c1-9678-4d2d-83e1-3f17daf16e7c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:46:07.156480 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-152e58c1-9678-4d2d-83e1-3f17daf16e7c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:46:07.158182 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-152e58c1-9678-4d2d-83e1-3f17daf16e7c None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:46:07.170391 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 623/1231] 10.222.0.33 () {56 vars in 1148 bytes} [Sun Oct 9 00:46:07 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:46:07.176674 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a4845bc-1b0a-4815-bfbc-cd1edbd33a17 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:07.243762 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:46:07.283688 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a4845bc-1b0a-4815-bfbc-cd1edbd33a17 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:07.319051 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6c3f3b5-9966-4b45-b5c0-e57be52f62a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:07.319744 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c3f3b5-9966-4b45-b5c0-e57be52f62a7 None admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:07.320025 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c3f3b5-9966-4b45-b5c0-e57be52f62a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:07.320300 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c3f3b5-9966-4b45-b5c0-e57be52f62a7 None admin] PATH_INFO: `/v3/projects/013c83476b1340fbb051061833c4a9c6/users/d25e240cb93643ed8122ccac9586936f/roles/228232e130c5412a80d774978175d28b` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:07.336202 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6c3f3b5-9966-4b45-b5c0-e57be52f62a7 None admin] RBAC: Authorizing `identity:create_grant(project_id=013c83476b1340fbb051061833c4a9c6, user_id=d25e240cb93643ed8122ccac9586936f, role_id=228232e130c5412a80d774978175d28b)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:46:07.337769 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6c3f3b5-9966-4b45-b5c0-e57be52f62a7 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:46:07.351923 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 609/1232] 10.222.0.33 () {58 vars in 1439 bytes} [Sun Oct 9 00:46:07 2022] PUT /identity/v3/projects/013c83476b1340fbb051061833c4a9c6/users/d25e240cb93643ed8122ccac9586936f/roles/228232e130c5412a80d774978175d28b => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:46:09.538872 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-796173bd-1ad8-4b52-8661-1030326f9cbc None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:09.539458 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-796173bd-1ad8-4b52-8661-1030326f9cbc None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:09.539458 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-796173bd-1ad8-4b52-8661-1030326f9cbc None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:09.540962 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 624/1233] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:46:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:46:09.549476 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9555cc52-d0ab-45fc-9456-8531d70fbfb7 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:09.550045 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9555cc52-d0ab-45fc-9456-8531d70fbfb7 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:09.550045 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9555cc52-d0ab-45fc-9456-8531d70fbfb7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:09.570771 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-9555cc52-d0ab-45fc-9456-8531d70fbfb7 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:46:09.572223 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-9555cc52-d0ab-45fc-9456-8531d70fbfb7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:46:09.646132 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 610/1234] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:46:09 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 09 00:46:09.928062 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-796173bd-1ad8-4b52-8661-1030326f9cbc None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:09.956885 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:46:10.041777 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-796173bd-1ad8-4b52-8661-1030326f9cbc None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:10.083977 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be8e5c73-c14a-4a69-ab6f-64e2f3de9409 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:10.084664 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be8e5c73-c14a-4a69-ab6f-64e2f3de9409 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:10.085010 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be8e5c73-c14a-4a69-ab6f-64e2f3de9409 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:10.085279 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be8e5c73-c14a-4a69-ab6f-64e2f3de9409 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:10.097994 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be8e5c73-c14a-4a69-ab6f-64e2f3de9409 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:46:10.104202 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-be8e5c73-c14a-4a69-ab6f-64e2f3de9409 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 09 00:46:10.104202 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:46:10.104202 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:46:10.104202 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:46:10.104202 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:46:10.104202 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:46:10.104202 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:46:10.104202 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:46:10.104202 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:46:10.104202 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:46:10.104202 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:46:10.104202 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:46:10.104202 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:46:10.104202 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:46:10.104202 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:46:10.104202 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:46:10.105331 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:46:10.105331 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:46:10.105331 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:46:10.105331 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:46:10.105331 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:46:10.105331 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:46:10.105331 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:46:10.105331 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:46:10.105331 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:46:10.105331 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:46:10.105331 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:46:10.105331 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:46:10.105331 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:46:10.105331 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:46:10.105331 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:46:10.105331 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:46:10.107112 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:46:10.107112 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:46:10.107112 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:46:10.107112 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:46:10.107112 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:46:10.107112 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:46:10.107112 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:46:10.107112 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:46:10.107112 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:46:10.107112 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:46:10.107112 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 09 00:46:10.107112 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:46:10.107112 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 625/1235] 10.222.0.33 () {56 vars in 1102 bytes} [Sun Oct 9 00:46:09 2022] GET /identity/v3/roles/service => generated 85 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:46:10.111948 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9555cc52-d0ab-45fc-9456-8531d70fbfb7 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:10.160545 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:46:10.220924 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9555cc52-d0ab-45fc-9456-8531d70fbfb7 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:10.258911 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4916da8b-c2cd-45c5-ad13-79c8a07095d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:10.259521 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4916da8b-c2cd-45c5-ad13-79c8a07095d2 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:10.259834 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4916da8b-c2cd-45c5-ad13-79c8a07095d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:10.260106 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4916da8b-c2cd-45c5-ad13-79c8a07095d2 None admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:10.260766 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4916da8b-c2cd-45c5-ad13-79c8a07095d2 None admin] RBAC: Adding query filter params (name=service) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:46:10.261063 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4916da8b-c2cd-45c5-ad13-79c8a07095d2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:46:10.262872 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4916da8b-c2cd-45c5-ad13-79c8a07095d2 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:46:10.273079 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 611/1236] 10.222.0.33 () {56 vars in 1103 bytes} [Sun Oct 9 00:46:10 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:46:10.279144 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be8e5c73-c14a-4a69-ab6f-64e2f3de9409 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:10.374707 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:46:10.392811 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be8e5c73-c14a-4a69-ab6f-64e2f3de9409 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:10.435521 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da09d82d-337b-43db-afab-ff4477b877a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:10.436176 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da09d82d-337b-43db-afab-ff4477b877a9 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:10.436452 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da09d82d-337b-43db-afab-ff4477b877a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:10.436724 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da09d82d-337b-43db-afab-ff4477b877a9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:10.448227 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da09d82d-337b-43db-afab-ff4477b877a9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:46:10.449917 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da09d82d-337b-43db-afab-ff4477b877a9 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:46:10.466675 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-da09d82d-337b-43db-afab-ff4477b877a9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:46:10.466675 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:46:10.466675 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:46:10.466675 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:46:10.466675 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:46:10.466675 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:46:10.466675 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:46:10.466675 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:46:10.466675 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:46:10.466675 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:46:10.466675 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:46:10.466675 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:46:10.466675 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:46:10.466675 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:46:10.466675 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:46:10.466675 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:46:10.467746 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:46:10.467746 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:46:10.467746 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:46:10.467746 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:46:10.467746 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:46:10.467746 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:46:10.467746 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:46:10.467746 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:46:10.467746 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:46:10.467746 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:46:10.467746 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:46:10.467746 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:46:10.467746 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:46:10.467746 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:46:10.467746 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:46:10.467746 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:46:10.467746 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:46:10.468659 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:46:10.468659 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:46:10.468659 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:46:10.468659 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 626/1237] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:46:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 09 00:46:10.474380 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4916da8b-c2cd-45c5-ad13-79c8a07095d2 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:10.585634 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4916da8b-c2cd-45c5-ad13-79c8a07095d2 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:10.595762 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:46:10.624530 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ed4227c-00c7-4f39-ba48-00df4d18b7a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:10.625140 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ed4227c-00c7-4f39-ba48-00df4d18b7a5 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:10.625417 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ed4227c-00c7-4f39-ba48-00df4d18b7a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:10.625680 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ed4227c-00c7-4f39-ba48-00df4d18b7a5 None admin] PATH_INFO: `/v3/domains` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:10.626315 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ed4227c-00c7-4f39-ba48-00df4d18b7a5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:46:10.626613 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ed4227c-00c7-4f39-ba48-00df4d18b7a5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:46:10.627997 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ed4227c-00c7-4f39-ba48-00df4d18b7a5 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:46:10.642666 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 612/1238] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:46:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:46:10.648832 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da09d82d-337b-43db-afab-ff4477b877a9 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:10.763961 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da09d82d-337b-43db-afab-ff4477b877a9 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:10.804954 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04f80680-cf47-440a-95f3-d2f3cebbfb40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:10.805747 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04f80680-cf47-440a-95f3-d2f3cebbfb40 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:10.806044 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04f80680-cf47-440a-95f3-d2f3cebbfb40 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:10.806390 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04f80680-cf47-440a-95f3-d2f3cebbfb40 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:10.809888 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:46:10.816244 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04f80680-cf47-440a-95f3-d2f3cebbfb40 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:46:10.818414 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04f80680-cf47-440a-95f3-d2f3cebbfb40 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:46:10.833521 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-04f80680-cf47-440a-95f3-d2f3cebbfb40 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 09 00:46:10.833521 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:46:10.833521 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:46:10.833521 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:46:10.833521 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:46:10.833521 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:46:10.833521 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:46:10.833521 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:46:10.833521 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:46:10.833521 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:46:10.833521 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:46:10.833521 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:46:10.833521 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 09 00:46:10.833521 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 09 00:46:10.833521 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 09 00:46:10.833521 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 09 00:46:10.834769 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:46:10.834769 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:46:10.834769 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 09 00:46:10.834769 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 09 00:46:10.834769 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 09 00:46:10.834769 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 09 00:46:10.834769 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:46:10.834769 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:46:10.834769 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:46:10.834769 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:46:10.834769 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:46:10.834769 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:46:10.834769 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:46:10.834769 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:46:10.834769 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:46:10.834769 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:46:10.834769 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:46:10.834769 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:46:10.835840 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:46:10.835840 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:46:10.835840 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 09 00:46:10.835840 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 09 00:46:10.835840 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 09 00:46:10.835840 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 09 00:46:10.835840 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 09 00:46:10.835840 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 09 00:46:10.835840 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 09 00:46:10.835840 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:46:10.835840 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 627/1239] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:46:10 2022] GET /identity/v3/users/placement => generated 87 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 09 00:46:10.842694 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ed4227c-00c7-4f39-ba48-00df4d18b7a5 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:10.947052 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ed4227c-00c7-4f39-ba48-00df4d18b7a5 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:10.981608 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5007b45-296d-49fe-bd37-ba82bac065bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:10.982659 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5007b45-296d-49fe-bd37-ba82bac065bb None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:10.982763 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5007b45-296d-49fe-bd37-ba82bac065bb None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:10.983013 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5007b45-296d-49fe-bd37-ba82bac065bb None admin] PATH_INFO: `/v3/users` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:10.983930 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5007b45-296d-49fe-bd37-ba82bac065bb None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:46:10.984256 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5007b45-296d-49fe-bd37-ba82bac065bb None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:46:10.986021 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5007b45-296d-49fe-bd37-ba82bac065bb None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:46:11.007621 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 613/1240] 10.222.0.33 () {56 vars in 1143 bytes} [Sun Oct 9 00:46:10 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:46:11.010162 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04f80680-cf47-440a-95f3-d2f3cebbfb40 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:11.129188 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04f80680-cf47-440a-95f3-d2f3cebbfb40 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:11.167656 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99be4ae0-005e-4473-b8be-03911fdb4c25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:11.168281 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99be4ae0-005e-4473-b8be-03911fdb4c25 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:11.168562 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99be4ae0-005e-4473-b8be-03911fdb4c25 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:11.168830 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99be4ae0-005e-4473-b8be-03911fdb4c25 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:11.235867 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:46:11.244480 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99be4ae0-005e-4473-b8be-03911fdb4c25 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:46:11.246004 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99be4ae0-005e-4473-b8be-03911fdb4c25 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:46:11.360363 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 628/1241] 10.222.0.33 () {58 vars in 1282 bytes} [Sun Oct 9 00:46:11 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 351 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 09 00:46:11.366115 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5007b45-296d-49fe-bd37-ba82bac065bb None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:11.394508 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:46:11.471056 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5007b45-296d-49fe-bd37-ba82bac065bb None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:11.507627 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93a9a3c3-a584-4721-aa03-1727477395b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:11.508273 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a9a3c3-a584-4721-aa03-1727477395b2 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:11.508556 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a9a3c3-a584-4721-aa03-1727477395b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:11.508832 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a9a3c3-a584-4721-aa03-1727477395b2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:11.520102 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93a9a3c3-a584-4721-aa03-1727477395b2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:46:11.521536 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93a9a3c3-a584-4721-aa03-1727477395b2 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:46:11.536278 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-93a9a3c3-a584-4721-aa03-1727477395b2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:46:11.536278 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:46:11.536278 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:46:11.536278 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:46:11.536278 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:46:11.536278 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:46:11.536278 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:46:11.536278 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:46:11.536278 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:46:11.536278 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:46:11.536278 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:46:11.536278 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:46:11.536278 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:46:11.536278 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:46:11.536278 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:46:11.536278 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:46:11.537411 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:46:11.537411 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:46:11.537411 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:46:11.537411 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:46:11.537411 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:46:11.537411 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:46:11.537411 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:46:11.537411 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:46:11.537411 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:46:11.537411 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:46:11.537411 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:46:11.537411 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:46:11.537411 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:46:11.537411 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:46:11.537411 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:46:11.537411 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:46:11.537411 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:46:11.538870 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:46:11.538870 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:46:11.538870 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:46:11.538870 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 614/1242] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:46:11 2022] GET /identity/v3/domains/Default => generated 87 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:46:11.543193 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99be4ae0-005e-4473-b8be-03911fdb4c25 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:11.646788 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:46:11.653186 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99be4ae0-005e-4473-b8be-03911fdb4c25 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:11.690188 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af0b678c-cc9f-4583-8feb-7be346767e89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:11.690852 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af0b678c-cc9f-4583-8feb-7be346767e89 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:11.691186 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af0b678c-cc9f-4583-8feb-7be346767e89 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:11.691517 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af0b678c-cc9f-4583-8feb-7be346767e89 None admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:11.692249 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af0b678c-cc9f-4583-8feb-7be346767e89 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:46:11.692554 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af0b678c-cc9f-4583-8feb-7be346767e89 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:46:11.694047 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af0b678c-cc9f-4583-8feb-7be346767e89 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:46:11.706046 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 629/1243] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:46:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:46:11.711755 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93a9a3c3-a584-4721-aa03-1727477395b2 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:11.797439 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:46:11.823980 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93a9a3c3-a584-4721-aa03-1727477395b2 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:11.858805 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-974d0d4c-1db6-46d2-ad68-5a58e8553d3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:11.859478 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974d0d4c-1db6-46d2-ad68-5a58e8553d3c None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:11.859787 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974d0d4c-1db6-46d2-ad68-5a58e8553d3c None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:11.860059 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974d0d4c-1db6-46d2-ad68-5a58e8553d3c None admin] PATH_INFO: `/v3/projects/service` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:11.867326 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-974d0d4c-1db6-46d2-ad68-5a58e8553d3c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:46:11.868977 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-974d0d4c-1db6-46d2-ad68-5a58e8553d3c None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:46:11.881649 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-974d0d4c-1db6-46d2-ad68-5a58e8553d3c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 09 00:46:11.881649 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:46:11.881649 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:46:11.881649 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:46:11.881649 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:46:11.881649 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:46:11.881649 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:46:11.881649 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:46:11.881649 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:46:11.881649 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:46:11.881649 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:46:11.881649 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:46:11.881649 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 09 00:46:11.881649 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 09 00:46:11.881649 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 09 00:46:11.882844 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 09 00:46:11.882844 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:46:11.882844 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:46:11.882844 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:46:11.882844 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:46:11.882844 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:46:11.882844 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:46:11.882844 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:46:11.882844 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:46:11.882844 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:46:11.882844 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:46:11.882844 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:46:11.882844 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:46:11.882844 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:46:11.882844 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:46:11.882844 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:46:11.882844 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:46:11.884121 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 09 00:46:11.884121 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 09 00:46:11.884121 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 09 00:46:11.884121 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 09 00:46:11.884121 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 09 00:46:11.884121 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 09 00:46:11.884121 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 09 00:46:11.884121 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:46:11.884121 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 615/1244] 10.222.0.33 () {56 vars in 1111 bytes} [Sun Oct 9 00:46:11 2022] GET /identity/v3/projects/service => generated 88 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:46:11.888899 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af0b678c-cc9f-4583-8feb-7be346767e89 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:12.003835 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af0b678c-cc9f-4583-8feb-7be346767e89 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:12.045238 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36e82a15-1362-4bed-99ce-86980043a909 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:12.045879 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36e82a15-1362-4bed-99ce-86980043a909 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:12.046461 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36e82a15-1362-4bed-99ce-86980043a909 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:12.046687 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36e82a15-1362-4bed-99ce-86980043a909 None admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:12.047448 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36e82a15-1362-4bed-99ce-86980043a909 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:46:12.047785 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36e82a15-1362-4bed-99ce-86980043a909 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:46:12.049278 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36e82a15-1362-4bed-99ce-86980043a909 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:46:12.061925 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 630/1245] 10.222.0.33 () {56 vars in 1148 bytes} [Sun Oct 9 00:46:11 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:46:12.068182 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-974d0d4c-1db6-46d2-ad68-5a58e8553d3c None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:12.180087 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-974d0d4c-1db6-46d2-ad68-5a58e8553d3c None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:12.216873 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d98f235-461c-441d-9f16-fa8e4fb5a4b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:12.217485 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d98f235-461c-441d-9f16-fa8e4fb5a4b4 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:12.217763 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d98f235-461c-441d-9f16-fa8e4fb5a4b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:12.218035 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d98f235-461c-441d-9f16-fa8e4fb5a4b4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:12.218759 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d98f235-461c-441d-9f16-fa8e4fb5a4b4 None admin] RBAC: Adding query filter params (role.id=228232e130c5412a80d774978175d28b, scope.project.id=013c83476b1340fbb051061833c4a9c6, user.id=d25e240cb93643ed8122ccac9586936f) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:46:12.219060 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d98f235-461c-441d-9f16-fa8e4fb5a4b4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:46:12.220509 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d98f235-461c-441d-9f16-fa8e4fb5a4b4 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:46:12.227194 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 616/1246] 10.222.0.33 () {56 vars in 1374 bytes} [Sun Oct 9 00:46:12 2022] GET /identity/v3/role_assignments?user.id=d25e240cb93643ed8122ccac9586936f&scope.project.id=013c83476b1340fbb051061833c4a9c6&role.id=228232e130c5412a80d774978175d28b => generated 608 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:46:14.696736 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ab3078-46d1-4939-8984-ce02c30e37a4 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:14.697367 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ab3078-46d1-4939-8984-ce02c30e37a4 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:14.697367 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ab3078-46d1-4939-8984-ce02c30e37a4 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:14.698773 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 631/1247] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:46:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:46:14.705050 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba6b1962-7009-4f21-8912-f9ff4917b44f None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:14.705567 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba6b1962-7009-4f21-8912-f9ff4917b44f None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:14.705669 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba6b1962-7009-4f21-8912-f9ff4917b44f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:14.711865 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:46:14.725720 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-ba6b1962-7009-4f21-8912-f9ff4917b44f None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:46:14.726895 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-ba6b1962-7009-4f21-8912-f9ff4917b44f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:46:14.807469 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 617/1248] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:46:14 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 09 00:46:15.079380 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8ab3078-46d1-4939-8984-ce02c30e37a4 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:15.098368 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:46:15.192608 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8ab3078-46d1-4939-8984-ce02c30e37a4 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:15.229508 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e129f5b9-7ed6-4be4-86cc-2b3bdd3ae768 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:15.230138 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e129f5b9-7ed6-4be4-86cc-2b3bdd3ae768 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:15.230417 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e129f5b9-7ed6-4be4-86cc-2b3bdd3ae768 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:15.230702 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e129f5b9-7ed6-4be4-86cc-2b3bdd3ae768 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:15.244476 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e129f5b9-7ed6-4be4-86cc-2b3bdd3ae768 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:46:15.250228 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-e129f5b9-7ed6-4be4-86cc-2b3bdd3ae768 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 09 00:46:15.250228 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:46:15.250228 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:46:15.250228 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:46:15.250228 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:46:15.250228 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:46:15.250228 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:46:15.250228 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:46:15.250228 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:46:15.250228 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:46:15.250228 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:46:15.250228 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:46:15.250228 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:46:15.250228 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:46:15.250228 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:46:15.250228 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:46:15.251275 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:46:15.251275 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:46:15.251275 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:46:15.251275 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:46:15.251275 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:46:15.251275 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:46:15.251275 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:46:15.251275 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:46:15.251275 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:46:15.251275 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:46:15.251275 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:46:15.251275 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:46:15.251275 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:46:15.251275 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:46:15.251275 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:46:15.251275 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:46:15.256829 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:46:15.256829 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:46:15.256829 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:46:15.256829 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:46:15.256829 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:46:15.256829 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:46:15.256829 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:46:15.256829 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:46:15.256829 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:46:15.256829 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:46:15.256829 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 09 00:46:15.256829 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:46:15.256829 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 632/1249] 10.222.0.33 () {56 vars in 1096 bytes} [Sun Oct 9 00:46:15 2022] GET /identity/v3/roles/admin => generated 83 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:46:15.258450 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba6b1962-7009-4f21-8912-f9ff4917b44f None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:15.372451 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba6b1962-7009-4f21-8912-f9ff4917b44f None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:15.405389 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:46:15.410895 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b391469e-828f-4336-b6ce-9f472458882e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:15.411522 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b391469e-828f-4336-b6ce-9f472458882e None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:15.411846 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b391469e-828f-4336-b6ce-9f472458882e None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:15.412123 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b391469e-828f-4336-b6ce-9f472458882e None admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:15.412799 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b391469e-828f-4336-b6ce-9f472458882e None admin] RBAC: Adding query filter params (name=admin) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:46:15.413103 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b391469e-828f-4336-b6ce-9f472458882e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:46:15.414507 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b391469e-828f-4336-b6ce-9f472458882e None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:46:15.424106 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 618/1250] 10.222.0.33 () {56 vars in 1099 bytes} [Sun Oct 9 00:46:15 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:46:15.430066 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e129f5b9-7ed6-4be4-86cc-2b3bdd3ae768 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:15.512903 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:46:15.537971 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e129f5b9-7ed6-4be4-86cc-2b3bdd3ae768 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:15.574611 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fae8b8d3-65d4-4fae-86d0-f05bd0b0dced None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:15.575817 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae8b8d3-65d4-4fae-86d0-f05bd0b0dced None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:15.576101 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae8b8d3-65d4-4fae-86d0-f05bd0b0dced None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:15.576382 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae8b8d3-65d4-4fae-86d0-f05bd0b0dced None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:15.587071 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fae8b8d3-65d4-4fae-86d0-f05bd0b0dced None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:46:15.588640 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fae8b8d3-65d4-4fae-86d0-f05bd0b0dced None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:46:15.604236 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-fae8b8d3-65d4-4fae-86d0-f05bd0b0dced None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:46:15.604236 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:46:15.604236 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:46:15.604236 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:46:15.604236 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:46:15.604236 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:46:15.604236 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:46:15.604236 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:46:15.604236 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:46:15.604236 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:46:15.604236 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:46:15.604236 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:46:15.604236 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:46:15.604236 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:46:15.604236 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:46:15.604236 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:46:15.605455 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:46:15.605455 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:46:15.605455 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:46:15.605455 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:46:15.605455 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:46:15.605455 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:46:15.605455 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:46:15.605455 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:46:15.605455 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:46:15.605455 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:46:15.605455 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:46:15.605455 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:46:15.605455 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:46:15.605455 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:46:15.605455 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:46:15.605455 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:46:15.605455 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:46:15.607081 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:46:15.607081 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:46:15.607081 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:46:15.607081 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 633/1251] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:46:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:46:15.612087 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b391469e-828f-4336-b6ce-9f472458882e None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:15.717272 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b391469e-828f-4336-b6ce-9f472458882e None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:15.751947 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a73ef9b-fa44-4696-afb6-28d2eb982417 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:15.752563 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a73ef9b-fa44-4696-afb6-28d2eb982417 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:15.752845 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a73ef9b-fa44-4696-afb6-28d2eb982417 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:15.753120 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a73ef9b-fa44-4696-afb6-28d2eb982417 None admin] PATH_INFO: `/v3/domains` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:15.753757 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a73ef9b-fa44-4696-afb6-28d2eb982417 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:46:15.754059 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a73ef9b-fa44-4696-afb6-28d2eb982417 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:46:15.755421 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a73ef9b-fa44-4696-afb6-28d2eb982417 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:46:15.773915 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 619/1252] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:46:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:46:15.775647 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fae8b8d3-65d4-4fae-86d0-f05bd0b0dced None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:15.887086 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fae8b8d3-65d4-4fae-86d0-f05bd0b0dced None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:15.913258 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:46:15.922214 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb4a0e38-8c7a-4a67-8171-d0df1ea8a75c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:15.922834 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb4a0e38-8c7a-4a67-8171-d0df1ea8a75c None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:15.923108 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb4a0e38-8c7a-4a67-8171-d0df1ea8a75c None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:15.923373 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb4a0e38-8c7a-4a67-8171-d0df1ea8a75c None admin] PATH_INFO: `/v3/users/placement` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:15.931262 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb4a0e38-8c7a-4a67-8171-d0df1ea8a75c None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:46:15.932730 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb4a0e38-8c7a-4a67-8171-d0df1ea8a75c None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:46:15.944126 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-eb4a0e38-8c7a-4a67-8171-d0df1ea8a75c None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 09 00:46:15.944126 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:46:15.944126 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:46:15.944126 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:46:15.944126 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:46:15.944126 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:46:15.944126 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:46:15.944126 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:46:15.944126 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:46:15.944126 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:46:15.944126 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:46:15.944126 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:46:15.944126 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 09 00:46:15.944126 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 09 00:46:15.944126 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 09 00:46:15.944126 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 09 00:46:15.945300 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:46:15.945300 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:46:15.945300 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 09 00:46:15.945300 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 09 00:46:15.945300 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 09 00:46:15.945300 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 09 00:46:15.945300 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:46:15.945300 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:46:15.945300 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:46:15.945300 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:46:15.945300 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:46:15.945300 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:46:15.945300 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:46:15.945300 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:46:15.945300 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:46:15.945300 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:46:15.945300 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:46:15.945300 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:46:15.946542 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:46:15.946542 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:46:15.946542 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 09 00:46:15.946542 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 09 00:46:15.946542 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 09 00:46:15.946542 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 09 00:46:15.946542 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 09 00:46:15.946542 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 09 00:46:15.946542 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 09 00:46:15.946542 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:46:15.946542 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 634/1253] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:46:15 2022] GET /identity/v3/users/placement => generated 87 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:46:15.951050 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a73ef9b-fa44-4696-afb6-28d2eb982417 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:16.006899 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:46:16.050806 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a73ef9b-fa44-4696-afb6-28d2eb982417 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:16.084414 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0815fb9-b268-475f-946f-8a0bbaa517cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:16.085013 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0815fb9-b268-475f-946f-8a0bbaa517cb None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:16.085289 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0815fb9-b268-475f-946f-8a0bbaa517cb None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:16.085558 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0815fb9-b268-475f-946f-8a0bbaa517cb None admin] PATH_INFO: `/v3/users` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:16.086265 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0815fb9-b268-475f-946f-8a0bbaa517cb None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:46:16.086565 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0815fb9-b268-475f-946f-8a0bbaa517cb None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:46:16.088004 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0815fb9-b268-475f-946f-8a0bbaa517cb None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:46:16.104547 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 620/1254] 10.222.0.33 () {56 vars in 1143 bytes} [Sun Oct 9 00:46:15 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:46:16.110808 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb4a0e38-8c7a-4a67-8171-d0df1ea8a75c None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:16.207019 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb4a0e38-8c7a-4a67-8171-d0df1ea8a75c None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:16.239129 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12719f4c-5720-45ec-9bc3-e4da0c0ea117 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:16.239739 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12719f4c-5720-45ec-9bc3-e4da0c0ea117 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:16.240020 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12719f4c-5720-45ec-9bc3-e4da0c0ea117 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:16.240289 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12719f4c-5720-45ec-9bc3-e4da0c0ea117 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:16.292228 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:46:16.305074 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12719f4c-5720-45ec-9bc3-e4da0c0ea117 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:46:16.306507 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12719f4c-5720-45ec-9bc3-e4da0c0ea117 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:46:16.399786 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 635/1255] 10.222.0.33 () {58 vars in 1282 bytes} [Sun Oct 9 00:46:16 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 289 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 09 00:46:16.405642 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0815fb9-b268-475f-946f-8a0bbaa517cb None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:16.501133 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0815fb9-b268-475f-946f-8a0bbaa517cb None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:16.513992 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:46:16.534202 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42cd211b-3a78-48f3-a07f-fda101b77640 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:16.534810 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42cd211b-3a78-48f3-a07f-fda101b77640 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:16.535086 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42cd211b-3a78-48f3-a07f-fda101b77640 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:16.535352 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42cd211b-3a78-48f3-a07f-fda101b77640 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:16.545008 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42cd211b-3a78-48f3-a07f-fda101b77640 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:46:16.546363 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42cd211b-3a78-48f3-a07f-fda101b77640 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:46:16.560941 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-42cd211b-3a78-48f3-a07f-fda101b77640 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:46:16.560941 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:46:16.560941 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:46:16.560941 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:46:16.560941 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:46:16.560941 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:46:16.560941 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:46:16.560941 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:46:16.560941 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:46:16.560941 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:46:16.560941 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:46:16.560941 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:46:16.560941 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:46:16.560941 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:46:16.560941 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:46:16.560941 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:46:16.562027 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:46:16.562027 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:46:16.562027 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:46:16.562027 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:46:16.562027 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:46:16.562027 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:46:16.562027 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:46:16.562027 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:46:16.562027 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:46:16.562027 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:46:16.562027 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:46:16.562027 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:46:16.562027 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:46:16.562027 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:46:16.562027 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:46:16.562027 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:46:16.562027 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:46:16.562900 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:46:16.562900 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:46:16.562900 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:46:16.562900 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 621/1256] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:46:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:46:16.567684 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12719f4c-5720-45ec-9bc3-e4da0c0ea117 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:16.651234 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:46:16.663796 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12719f4c-5720-45ec-9bc3-e4da0c0ea117 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:16.697032 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b683db3-eb57-428b-96e5-d99dd1b9c20c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:16.697614 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b683db3-eb57-428b-96e5-d99dd1b9c20c None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:16.697891 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b683db3-eb57-428b-96e5-d99dd1b9c20c None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:16.698162 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b683db3-eb57-428b-96e5-d99dd1b9c20c None admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:16.698784 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b683db3-eb57-428b-96e5-d99dd1b9c20c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:46:16.699081 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b683db3-eb57-428b-96e5-d99dd1b9c20c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:46:16.701016 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b683db3-eb57-428b-96e5-d99dd1b9c20c None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:46:16.711822 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 636/1257] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:46:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:46:16.717367 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42cd211b-3a78-48f3-a07f-fda101b77640 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:16.823475 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42cd211b-3a78-48f3-a07f-fda101b77640 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:16.856935 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24814357-e9e3-4c14-ad3d-103946ba6c36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:16.857551 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24814357-e9e3-4c14-ad3d-103946ba6c36 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:16.857828 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24814357-e9e3-4c14-ad3d-103946ba6c36 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:16.858097 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24814357-e9e3-4c14-ad3d-103946ba6c36 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:16.864415 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24814357-e9e3-4c14-ad3d-103946ba6c36 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:46:16.865835 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24814357-e9e3-4c14-ad3d-103946ba6c36 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:46:16.877207 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-24814357-e9e3-4c14-ad3d-103946ba6c36 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 09 00:46:16.877207 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:46:16.877207 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:46:16.877207 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:46:16.877207 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:46:16.877207 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:46:16.877207 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:46:16.877207 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:46:16.877207 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:46:16.877207 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:46:16.877207 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:46:16.877207 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:46:16.877207 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 09 00:46:16.877207 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 09 00:46:16.877207 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 09 00:46:16.878251 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 09 00:46:16.878251 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:46:16.878251 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:46:16.878251 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:46:16.878251 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:46:16.878251 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:46:16.878251 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:46:16.878251 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:46:16.878251 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:46:16.878251 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:46:16.878251 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:46:16.878251 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:46:16.878251 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:46:16.878251 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:46:16.878251 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:46:16.878251 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:46:16.878251 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:46:16.879519 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 09 00:46:16.879519 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 09 00:46:16.879519 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 09 00:46:16.879519 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 09 00:46:16.879519 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 09 00:46:16.879519 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 09 00:46:16.879519 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 09 00:46:16.879519 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:46:16.879519 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 622/1258] 10.222.0.33 () {56 vars in 1111 bytes} [Sun Oct 9 00:46:16 2022] GET /identity/v3/projects/service => generated 88 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 09 00:46:16.884144 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b683db3-eb57-428b-96e5-d99dd1b9c20c None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:16.980185 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b683db3-eb57-428b-96e5-d99dd1b9c20c None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:17.012811 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43f5d56a-bda7-487b-8a1e-83756b5aafb6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:17.013395 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f5d56a-bda7-487b-8a1e-83756b5aafb6 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:17.013673 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f5d56a-bda7-487b-8a1e-83756b5aafb6 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:17.013944 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f5d56a-bda7-487b-8a1e-83756b5aafb6 None admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:17.014616 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43f5d56a-bda7-487b-8a1e-83756b5aafb6 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:46:17.014920 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43f5d56a-bda7-487b-8a1e-83756b5aafb6 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:46:17.016364 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43f5d56a-bda7-487b-8a1e-83756b5aafb6 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:46:17.027784 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 637/1259] 10.222.0.33 () {56 vars in 1148 bytes} [Sun Oct 9 00:46:16 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:46:17.033747 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24814357-e9e3-4c14-ad3d-103946ba6c36 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:17.060489 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:46:17.131867 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24814357-e9e3-4c14-ad3d-103946ba6c36 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:17.164312 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88f93990-ef3c-4fe8-965a-313b2d6ede5f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:17.164903 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88f93990-ef3c-4fe8-965a-313b2d6ede5f None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:17.165180 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88f93990-ef3c-4fe8-965a-313b2d6ede5f None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:17.165452 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88f93990-ef3c-4fe8-965a-313b2d6ede5f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:17.166157 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88f93990-ef3c-4fe8-965a-313b2d6ede5f None admin] RBAC: Adding query filter params (role.id=092036ee46ac427baf4d56468575f697, scope.project.id=013c83476b1340fbb051061833c4a9c6, user.id=d25e240cb93643ed8122ccac9586936f) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:46:17.166460 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88f93990-ef3c-4fe8-965a-313b2d6ede5f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:46:17.167896 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88f93990-ef3c-4fe8-965a-313b2d6ede5f None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:46:17.173382 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 623/1260] 10.222.0.33 () {56 vars in 1374 bytes} [Sun Oct 9 00:46:17 2022] GET /identity/v3/role_assignments?user.id=d25e240cb93643ed8122ccac9586936f&scope.project.id=013c83476b1340fbb051061833c4a9c6&role.id=092036ee46ac427baf4d56468575f697 => generated 259 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:46:19.281481 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f4c541b-4bce-4831-b94d-55f65e1fb2f2 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:19.282026 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f4c541b-4bce-4831-b94d-55f65e1fb2f2 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:19.282026 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f4c541b-4bce-4831-b94d-55f65e1fb2f2 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:19.283370 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 638/1261] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:46:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:46:19.288387 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db92459c-2235-4095-a793-78bd33b163c8 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:19.288815 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db92459c-2235-4095-a793-78bd33b163c8 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:19.288939 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db92459c-2235-4095-a793-78bd33b163c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:19.307569 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-db92459c-2235-4095-a793-78bd33b163c8 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:46:19.308772 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-db92459c-2235-4095-a793-78bd33b163c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:46:19.372780 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 624/1262] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:46:19 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 09 00:46:19.638680 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f4c541b-4bce-4831-b94d-55f65e1fb2f2 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:19.649358 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:46:19.738314 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f4c541b-4bce-4831-b94d-55f65e1fb2f2 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:19.771722 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd016ada-0685-43e7-971e-5ef2898b0788 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:19.772348 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd016ada-0685-43e7-971e-5ef2898b0788 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:19.772649 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd016ada-0685-43e7-971e-5ef2898b0788 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:19.773015 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd016ada-0685-43e7-971e-5ef2898b0788 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:19.786052 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd016ada-0685-43e7-971e-5ef2898b0788 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:46:19.791103 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-cd016ada-0685-43e7-971e-5ef2898b0788 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 09 00:46:19.791103 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:46:19.791103 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:46:19.791103 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:46:19.791103 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:46:19.791103 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:46:19.791103 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:46:19.791103 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:46:19.791103 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:46:19.791103 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:46:19.791103 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:46:19.791103 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:46:19.791103 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:46:19.791103 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:46:19.791103 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:46:19.791103 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:46:19.792276 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:46:19.792276 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:46:19.792276 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:46:19.792276 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:46:19.792276 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:46:19.792276 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:46:19.792276 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:46:19.792276 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:46:19.792276 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:46:19.792276 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:46:19.792276 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:46:19.792276 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:46:19.792276 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:46:19.792276 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:46:19.792276 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:46:19.792276 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:46:19.793308 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:46:19.793308 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:46:19.793308 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:46:19.793308 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:46:19.793308 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:46:19.793308 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:46:19.793308 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:46:19.793308 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:46:19.793308 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:46:19.793308 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:46:19.793308 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 09 00:46:19.793308 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:46:19.793308 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 639/1263] 10.222.0.33 () {56 vars in 1096 bytes} [Sun Oct 9 00:46:19 2022] GET /identity/v3/roles/admin => generated 83 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:46:19.799107 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db92459c-2235-4095-a793-78bd33b163c8 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:19.821442 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:46:19.904152 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db92459c-2235-4095-a793-78bd33b163c8 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:19.936422 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1415c388-9aa4-4f7a-833e-a83ec29b4b0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:19.937015 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1415c388-9aa4-4f7a-833e-a83ec29b4b0b None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:19.937297 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1415c388-9aa4-4f7a-833e-a83ec29b4b0b None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:19.937570 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1415c388-9aa4-4f7a-833e-a83ec29b4b0b None admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:19.938218 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1415c388-9aa4-4f7a-833e-a83ec29b4b0b None admin] RBAC: Adding query filter params (name=admin) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:46:19.938521 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1415c388-9aa4-4f7a-833e-a83ec29b4b0b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:46:19.939961 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1415c388-9aa4-4f7a-833e-a83ec29b4b0b None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:46:19.947780 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 625/1264] 10.222.0.33 () {56 vars in 1099 bytes} [Sun Oct 9 00:46:19 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:46:19.954009 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd016ada-0685-43e7-971e-5ef2898b0788 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:20.018238 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:46:20.051688 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd016ada-0685-43e7-971e-5ef2898b0788 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:20.086111 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c70244e2-5648-49cf-ae20-8398f5645e42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:20.086732 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c70244e2-5648-49cf-ae20-8398f5645e42 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:20.087012 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c70244e2-5648-49cf-ae20-8398f5645e42 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:20.087283 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c70244e2-5648-49cf-ae20-8398f5645e42 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:20.097639 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c70244e2-5648-49cf-ae20-8398f5645e42 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:46:20.099072 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c70244e2-5648-49cf-ae20-8398f5645e42 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:46:20.113187 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-c70244e2-5648-49cf-ae20-8398f5645e42 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:46:20.113187 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:46:20.113187 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:46:20.113187 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:46:20.113187 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:46:20.113187 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:46:20.113187 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:46:20.113187 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:46:20.113187 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:46:20.113187 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:46:20.113187 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:46:20.113187 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:46:20.113187 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:46:20.113187 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:46:20.113187 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:46:20.113187 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:46:20.114272 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:46:20.114272 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:46:20.114272 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:46:20.114272 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:46:20.114272 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:46:20.114272 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:46:20.114272 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:46:20.114272 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:46:20.114272 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:46:20.114272 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:46:20.114272 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:46:20.114272 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:46:20.114272 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:46:20.114272 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:46:20.114272 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:46:20.114272 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:46:20.114272 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:46:20.115166 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:46:20.115166 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:46:20.115166 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:46:20.115166 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 640/1265] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:46:19 2022] GET /identity/v3/domains/Default => generated 87 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:46:20.119950 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1415c388-9aa4-4f7a-833e-a83ec29b4b0b None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:20.197395 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:46:20.216800 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1415c388-9aa4-4f7a-833e-a83ec29b4b0b None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:20.248879 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a990f129-9596-494f-9816-45a357d4f476 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:20.249473 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a990f129-9596-494f-9816-45a357d4f476 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:20.249757 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a990f129-9596-494f-9816-45a357d4f476 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:20.250029 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a990f129-9596-494f-9816-45a357d4f476 None admin] PATH_INFO: `/v3/domains` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:20.250654 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a990f129-9596-494f-9816-45a357d4f476 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:46:20.250957 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a990f129-9596-494f-9816-45a357d4f476 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:46:20.252356 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a990f129-9596-494f-9816-45a357d4f476 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:46:20.264351 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 626/1266] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:46:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:46:20.269828 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c70244e2-5648-49cf-ae20-8398f5645e42 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:20.366437 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c70244e2-5648-49cf-ae20-8398f5645e42 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:20.385893 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:46:20.400594 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ef44108-871f-48e4-8630-e8b54afbc7b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:20.401201 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ef44108-871f-48e4-8630-e8b54afbc7b2 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:20.401481 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ef44108-871f-48e4-8630-e8b54afbc7b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:20.401757 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ef44108-871f-48e4-8630-e8b54afbc7b2 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:20.408980 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ef44108-871f-48e4-8630-e8b54afbc7b2 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:46:20.410405 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ef44108-871f-48e4-8630-e8b54afbc7b2 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:46:20.421240 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-2ef44108-871f-48e4-8630-e8b54afbc7b2 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 09 00:46:20.421240 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:46:20.421240 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:46:20.421240 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:46:20.421240 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:46:20.421240 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:46:20.421240 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:46:20.421240 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:46:20.421240 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:46:20.421240 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:46:20.421240 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:46:20.421240 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:46:20.421240 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 09 00:46:20.421240 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 09 00:46:20.421240 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 09 00:46:20.421240 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 09 00:46:20.422586 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:46:20.422586 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:46:20.422586 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 09 00:46:20.422586 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 09 00:46:20.422586 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 09 00:46:20.422586 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 09 00:46:20.422586 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:46:20.422586 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:46:20.422586 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:46:20.422586 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:46:20.422586 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:46:20.422586 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:46:20.422586 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:46:20.422586 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:46:20.422586 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:46:20.422586 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:46:20.422586 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:46:20.422586 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:46:20.423909 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:46:20.423909 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:46:20.423909 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 09 00:46:20.423909 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 09 00:46:20.423909 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 09 00:46:20.423909 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 09 00:46:20.423909 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 09 00:46:20.423909 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 09 00:46:20.423909 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 09 00:46:20.423909 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:46:20.423909 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 641/1267] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:46:20 2022] GET /identity/v3/users/placement => generated 87 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:46:20.428256 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a990f129-9596-494f-9816-45a357d4f476 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:20.524204 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a990f129-9596-494f-9816-45a357d4f476 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:20.556122 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3365bb6f-c5a1-4bc3-8e18-f0ad9804603a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:20.556710 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3365bb6f-c5a1-4bc3-8e18-f0ad9804603a None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:20.556992 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3365bb6f-c5a1-4bc3-8e18-f0ad9804603a None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:20.557267 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3365bb6f-c5a1-4bc3-8e18-f0ad9804603a None admin] PATH_INFO: `/v3/users` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:20.557975 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3365bb6f-c5a1-4bc3-8e18-f0ad9804603a None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:46:20.558279 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3365bb6f-c5a1-4bc3-8e18-f0ad9804603a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:46:20.560251 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3365bb6f-c5a1-4bc3-8e18-f0ad9804603a None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:46:20.568531 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:46:20.577431 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 627/1268] 10.222.0.33 () {56 vars in 1143 bytes} [Sun Oct 9 00:46:20 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:46:20.582952 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ef44108-871f-48e4-8630-e8b54afbc7b2 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:20.678199 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ef44108-871f-48e4-8630-e8b54afbc7b2 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:20.710884 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b2fbd02-e7b2-49e9-beb7-43e8b575954a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:20.711484 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b2fbd02-e7b2-49e9-beb7-43e8b575954a None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:20.711798 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b2fbd02-e7b2-49e9-beb7-43e8b575954a None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:20.712075 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b2fbd02-e7b2-49e9-beb7-43e8b575954a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:20.722167 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b2fbd02-e7b2-49e9-beb7-43e8b575954a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:46:20.723587 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b2fbd02-e7b2-49e9-beb7-43e8b575954a None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:46:20.738395 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-1b2fbd02-e7b2-49e9-beb7-43e8b575954a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:46:20.738395 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:46:20.738395 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:46:20.738395 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:46:20.738395 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:46:20.738395 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:46:20.738395 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:46:20.738395 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:46:20.738395 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:46:20.738395 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:46:20.738395 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:46:20.738395 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:46:20.738395 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:46:20.738395 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:46:20.738395 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:46:20.738395 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:46:20.739512 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:46:20.739512 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:46:20.739512 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:46:20.739512 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:46:20.739512 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:46:20.739512 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:46:20.739512 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:46:20.739512 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:46:20.739512 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:46:20.739512 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:46:20.739512 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:46:20.739512 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:46:20.739512 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:46:20.739512 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:46:20.739512 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:46:20.739512 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:46:20.739512 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:46:20.740786 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:46:20.740786 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:46:20.740786 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:46:20.740786 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 642/1269] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:46:20 2022] GET /identity/v3/domains/Default => generated 87 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:46:20.744964 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3365bb6f-c5a1-4bc3-8e18-f0ad9804603a None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:20.847360 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3365bb6f-c5a1-4bc3-8e18-f0ad9804603a None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:20.880219 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd128416-ce25-41dd-85bb-59b8005b2849 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:20.880812 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd128416-ce25-41dd-85bb-59b8005b2849 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:20.881090 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd128416-ce25-41dd-85bb-59b8005b2849 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:20.881360 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd128416-ce25-41dd-85bb-59b8005b2849 None admin] PATH_INFO: `/v3/domains` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:20.881985 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd128416-ce25-41dd-85bb-59b8005b2849 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:46:20.882283 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd128416-ce25-41dd-85bb-59b8005b2849 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:46:20.883656 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd128416-ce25-41dd-85bb-59b8005b2849 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:46:20.894489 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 628/1270] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:46:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:46:20.900166 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b2fbd02-e7b2-49e9-beb7-43e8b575954a None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:20.932199 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:46:20.999341 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b2fbd02-e7b2-49e9-beb7-43e8b575954a None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:21.031699 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0c516eb-ea85-43e2-8db3-cbfaa234e3ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:21.032303 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c516eb-ea85-43e2-8db3-cbfaa234e3ca None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:21.032580 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c516eb-ea85-43e2-8db3-cbfaa234e3ca None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:21.032851 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c516eb-ea85-43e2-8db3-cbfaa234e3ca None admin] PATH_INFO: `/v3/projects/service` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:21.039243 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0c516eb-ea85-43e2-8db3-cbfaa234e3ca None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:46:21.040738 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0c516eb-ea85-43e2-8db3-cbfaa234e3ca None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:46:21.052610 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-e0c516eb-ea85-43e2-8db3-cbfaa234e3ca None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 09 00:46:21.052610 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:46:21.052610 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:46:21.052610 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:46:21.052610 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:46:21.052610 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:46:21.052610 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:46:21.052610 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:46:21.052610 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:46:21.052610 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:46:21.052610 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:46:21.052610 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:46:21.052610 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 09 00:46:21.052610 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 09 00:46:21.052610 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 09 00:46:21.053671 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 09 00:46:21.053671 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:46:21.053671 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:46:21.053671 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:46:21.053671 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:46:21.053671 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:46:21.053671 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:46:21.053671 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:46:21.053671 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:46:21.053671 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:46:21.053671 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:46:21.053671 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:46:21.053671 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:46:21.053671 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:46:21.053671 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:46:21.053671 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:46:21.053671 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:46:21.054586 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 09 00:46:21.054586 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 09 00:46:21.054586 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 09 00:46:21.054586 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 09 00:46:21.054586 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 09 00:46:21.054586 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 09 00:46:21.054586 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 09 00:46:21.054586 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:46:21.054586 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 643/1271] 10.222.0.33 () {56 vars in 1111 bytes} [Sun Oct 9 00:46:20 2022] GET /identity/v3/projects/service => generated 88 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:46:21.059411 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd128416-ce25-41dd-85bb-59b8005b2849 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:21.116416 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:46:21.156672 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd128416-ce25-41dd-85bb-59b8005b2849 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:21.189169 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecda0f20-79d7-4958-b1e4-c440171d71ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:21.189758 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecda0f20-79d7-4958-b1e4-c440171d71ba None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:21.190038 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecda0f20-79d7-4958-b1e4-c440171d71ba None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:21.190311 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecda0f20-79d7-4958-b1e4-c440171d71ba None admin] PATH_INFO: `/v3/projects` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:21.190983 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecda0f20-79d7-4958-b1e4-c440171d71ba None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:46:21.191286 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecda0f20-79d7-4958-b1e4-c440171d71ba None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:46:21.192727 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecda0f20-79d7-4958-b1e4-c440171d71ba None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:46:21.204143 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 629/1272] 10.222.0.33 () {56 vars in 1148 bytes} [Sun Oct 9 00:46:21 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:46:21.210135 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0c516eb-ea85-43e2-8db3-cbfaa234e3ca None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:21.297938 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:46:21.307030 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0c516eb-ea85-43e2-8db3-cbfaa234e3ca None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:21.338614 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4fc7ed1-95be-4f41-9290-4262170cfbac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:21.339274 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4fc7ed1-95be-4f41-9290-4262170cfbac None admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:21.339562 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4fc7ed1-95be-4f41-9290-4262170cfbac None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:21.339874 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4fc7ed1-95be-4f41-9290-4262170cfbac None admin] PATH_INFO: `/v3/projects/013c83476b1340fbb051061833c4a9c6/users/d25e240cb93643ed8122ccac9586936f/roles/092036ee46ac427baf4d56468575f697` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:21.342306 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4fc7ed1-95be-4f41-9290-4262170cfbac None admin] RBAC: Authorizing `identity:create_grant(project_id=013c83476b1340fbb051061833c4a9c6, user_id=d25e240cb93643ed8122ccac9586936f, role_id=092036ee46ac427baf4d56468575f697)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:46:21.343776 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4fc7ed1-95be-4f41-9290-4262170cfbac None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:46:21.355571 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 644/1273] 10.222.0.33 () {58 vars in 1439 bytes} [Sun Oct 9 00:46:21 2022] PUT /identity/v3/projects/013c83476b1340fbb051061833c4a9c6/users/d25e240cb93643ed8122ccac9586936f/roles/092036ee46ac427baf4d56468575f697 => generated 0 bytes in 146 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:46:23.458053 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9904094d-ba94-4083-9543-797ee08c2cd8 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:23.458569 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9904094d-ba94-4083-9543-797ee08c2cd8 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:23.458678 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9904094d-ba94-4083-9543-797ee08c2cd8 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:23.459938 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 630/1274] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:46:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:46:23.464992 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dfbbad0-5421-4c1a-9e3a-294a6e27340d None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:23.465365 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dfbbad0-5421-4c1a-9e3a-294a6e27340d None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:23.465541 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dfbbad0-5421-4c1a-9e3a-294a6e27340d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:23.483400 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-3dfbbad0-5421-4c1a-9e3a-294a6e27340d None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:46:23.484593 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-3dfbbad0-5421-4c1a-9e3a-294a6e27340d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:46:23.549610 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 645/1275] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:46:23 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 09 00:46:23.826095 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9904094d-ba94-4083-9543-797ee08c2cd8 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:23.924426 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9904094d-ba94-4083-9543-797ee08c2cd8 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:23.941702 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:46:23.959099 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8510cf5-1897-4033-8adf-a37e885576a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:23.959750 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8510cf5-1897-4033-8adf-a37e885576a3 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:23.960027 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8510cf5-1897-4033-8adf-a37e885576a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:23.960293 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8510cf5-1897-4033-8adf-a37e885576a3 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:23.970518 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8510cf5-1897-4033-8adf-a37e885576a3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 09 00:46:23.975280 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-b8510cf5-1897-4033-8adf-a37e885576a3 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 09 00:46:23.975280 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:46:23.975280 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:46:23.975280 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:46:23.975280 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:46:23.975280 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:46:23.975280 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:46:23.975280 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:46:23.975280 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:46:23.975280 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:46:23.975280 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:46:23.975280 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:46:23.975280 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 09 00:46:23.975280 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 09 00:46:23.975280 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 09 00:46:23.975280 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 09 00:46:23.976377 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 09 00:46:23.976377 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 09 00:46:23.976377 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 09 00:46:23.976377 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 09 00:46:23.976377 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:46:23.976377 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:46:23.976377 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:46:23.976377 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:46:23.976377 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:46:23.976377 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:46:23.976377 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:46:23.976377 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:46:23.976377 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:46:23.976377 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:46:23.976377 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:46:23.976377 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:46:23.977401 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:46:23.977401 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:46:23.977401 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:46:23.977401 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:46:23.977401 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 09 00:46:23.977401 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 09 00:46:23.977401 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 09 00:46:23.977401 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 09 00:46:23.977401 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 09 00:46:23.977401 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 09 00:46:23.977401 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 09 00:46:23.977401 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:46:23.977401 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 631/1276] 10.222.0.33 () {56 vars in 1096 bytes} [Sun Oct 9 00:46:23 2022] GET /identity/v3/roles/admin => generated 83 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:46:23.982460 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dfbbad0-5421-4c1a-9e3a-294a6e27340d None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:24.060925 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:46:24.080363 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dfbbad0-5421-4c1a-9e3a-294a6e27340d None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:24.113062 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-509e6ebe-268d-49c0-a48d-aa99d04f90db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:24.113651 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-509e6ebe-268d-49c0-a48d-aa99d04f90db None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:24.113927 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-509e6ebe-268d-49c0-a48d-aa99d04f90db None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:24.114198 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-509e6ebe-268d-49c0-a48d-aa99d04f90db None admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:24.114843 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-509e6ebe-268d-49c0-a48d-aa99d04f90db None admin] RBAC: Adding query filter params (name=admin) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:46:24.115140 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-509e6ebe-268d-49c0-a48d-aa99d04f90db None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:46:24.116515 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-509e6ebe-268d-49c0-a48d-aa99d04f90db None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:46:24.124784 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 646/1277] 10.222.0.33 () {56 vars in 1099 bytes} [Sun Oct 9 00:46:23 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:46:24.130675 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8510cf5-1897-4033-8adf-a37e885576a3 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:24.225206 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8510cf5-1897-4033-8adf-a37e885576a3 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:24.257047 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a717c83-bd14-470a-92ab-f4a6fee847a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:24.257654 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a717c83-bd14-470a-92ab-f4a6fee847a1 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:24.257925 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a717c83-bd14-470a-92ab-f4a6fee847a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:24.258192 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a717c83-bd14-470a-92ab-f4a6fee847a1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:24.268156 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a717c83-bd14-470a-92ab-f4a6fee847a1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:46:24.269595 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a717c83-bd14-470a-92ab-f4a6fee847a1 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:46:24.283440 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-0a717c83-bd14-470a-92ab-f4a6fee847a1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:46:24.283440 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:46:24.283440 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:46:24.283440 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:46:24.283440 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:46:24.283440 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:46:24.283440 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:46:24.283440 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:46:24.283440 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:46:24.283440 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:46:24.283440 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:46:24.283440 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:46:24.283440 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:46:24.283440 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:46:24.283440 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:46:24.283440 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:46:24.284592 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:46:24.284592 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:46:24.284592 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:46:24.284592 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:46:24.284592 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:46:24.284592 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:46:24.284592 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:46:24.284592 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:46:24.284592 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:46:24.284592 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:46:24.284592 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:46:24.284592 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:46:24.284592 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:46:24.284592 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:46:24.284592 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:46:24.284592 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:46:24.284592 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:46:24.285664 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:46:24.285664 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:46:24.285664 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:46:24.285664 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 632/1278] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:46:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:46:24.290051 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-509e6ebe-268d-49c0-a48d-aa99d04f90db None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:24.386739 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-509e6ebe-268d-49c0-a48d-aa99d04f90db None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:24.418048 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aec2ac3-cf55-42f4-a36a-a7368d5a2cef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:24.418626 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aec2ac3-cf55-42f4-a36a-a7368d5a2cef None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:24.419432 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aec2ac3-cf55-42f4-a36a-a7368d5a2cef None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:24.419732 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aec2ac3-cf55-42f4-a36a-a7368d5a2cef None admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:24.420352 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aec2ac3-cf55-42f4-a36a-a7368d5a2cef None admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:46:24.420644 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aec2ac3-cf55-42f4-a36a-a7368d5a2cef None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:46:24.421973 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aec2ac3-cf55-42f4-a36a-a7368d5a2cef None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:46:24.433338 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 647/1279] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:46:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:46:24.439538 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a717c83-bd14-470a-92ab-f4a6fee847a1 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:24.468555 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:46:24.534955 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a717c83-bd14-470a-92ab-f4a6fee847a1 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:24.567546 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03d48ea6-e4bd-4067-9f5e-d4191b162bbb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:24.568183 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d48ea6-e4bd-4067-9f5e-d4191b162bbb None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:24.568459 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d48ea6-e4bd-4067-9f5e-d4191b162bbb None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:24.568728 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d48ea6-e4bd-4067-9f5e-d4191b162bbb None admin] PATH_INFO: `/v3/users/placement` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:24.576345 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03d48ea6-e4bd-4067-9f5e-d4191b162bbb None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:46:24.577750 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03d48ea6-e4bd-4067-9f5e-d4191b162bbb None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:46:24.588416 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-03d48ea6-e4bd-4067-9f5e-d4191b162bbb None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 09 00:46:24.588416 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:46:24.588416 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:46:24.588416 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:46:24.588416 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:46:24.588416 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:46:24.588416 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:46:24.588416 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:46:24.588416 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:46:24.588416 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:46:24.588416 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:46:24.588416 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:46:24.588416 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 09 00:46:24.588416 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 09 00:46:24.588416 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 09 00:46:24.588416 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 09 00:46:24.589484 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:46:24.589484 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:46:24.589484 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 09 00:46:24.589484 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 09 00:46:24.589484 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 09 00:46:24.589484 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 09 00:46:24.589484 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:46:24.589484 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:46:24.589484 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:46:24.589484 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:46:24.589484 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:46:24.589484 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:46:24.589484 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:46:24.589484 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:46:24.589484 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:46:24.589484 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:46:24.589484 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:46:24.589484 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:46:24.591521 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:46:24.591521 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:46:24.591521 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 09 00:46:24.591521 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 09 00:46:24.591521 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 09 00:46:24.591521 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 09 00:46:24.591521 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 09 00:46:24.591521 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 09 00:46:24.591521 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 09 00:46:24.591521 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:46:24.591521 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 633/1280] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:46:24 2022] GET /identity/v3/users/placement => generated 87 bytes in 150 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:46:24.595622 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aec2ac3-cf55-42f4-a36a-a7368d5a2cef None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:24.599293 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:46:24.691798 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aec2ac3-cf55-42f4-a36a-a7368d5a2cef None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:24.724604 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b8b2390-c673-46e9-b767-c9b0638812c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:24.725193 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b8b2390-c673-46e9-b767-c9b0638812c9 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:24.725470 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b8b2390-c673-46e9-b767-c9b0638812c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:24.725742 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b8b2390-c673-46e9-b767-c9b0638812c9 None admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:24.726449 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b8b2390-c673-46e9-b767-c9b0638812c9 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:46:24.726748 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b8b2390-c673-46e9-b767-c9b0638812c9 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:46:24.728190 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b8b2390-c673-46e9-b767-c9b0638812c9 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:46:24.744444 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 648/1281] 10.222.0.33 () {56 vars in 1143 bytes} [Sun Oct 9 00:46:24 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:46:24.750687 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03d48ea6-e4bd-4067-9f5e-d4191b162bbb None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:24.839806 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:46:24.850000 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03d48ea6-e4bd-4067-9f5e-d4191b162bbb None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:24.881794 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e6194ab-3523-4854-a432-8255bbcde434 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:24.882379 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e6194ab-3523-4854-a432-8255bbcde434 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:24.882657 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e6194ab-3523-4854-a432-8255bbcde434 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:24.882933 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e6194ab-3523-4854-a432-8255bbcde434 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:24.946611 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e6194ab-3523-4854-a432-8255bbcde434 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:46:24.948008 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e6194ab-3523-4854-a432-8255bbcde434 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:46:25.031842 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:46:25.041862 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 634/1282] 10.222.0.33 () {58 vars in 1282 bytes} [Sun Oct 9 00:46:24 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 292 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 09 00:46:25.048512 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b8b2390-c673-46e9-b767-c9b0638812c9 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:25.111438 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:46:25.148854 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b8b2390-c673-46e9-b767-c9b0638812c9 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:25.181676 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d8dd188-3973-47cb-9a4e-96bf8ce88dd2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:25.182292 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d8dd188-3973-47cb-9a4e-96bf8ce88dd2 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:25.182570 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d8dd188-3973-47cb-9a4e-96bf8ce88dd2 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:25.182842 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d8dd188-3973-47cb-9a4e-96bf8ce88dd2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:25.192373 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d8dd188-3973-47cb-9a4e-96bf8ce88dd2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:46:25.193800 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d8dd188-3973-47cb-9a4e-96bf8ce88dd2 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:46:25.208374 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-0d8dd188-3973-47cb-9a4e-96bf8ce88dd2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:46:25.208374 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:46:25.208374 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:46:25.208374 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:46:25.208374 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:46:25.208374 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:46:25.208374 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:46:25.208374 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:46:25.208374 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:46:25.208374 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:46:25.208374 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:46:25.208374 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:46:25.208374 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 09 00:46:25.208374 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 09 00:46:25.208374 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 09 00:46:25.208374 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 09 00:46:25.209479 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:46:25.209479 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:46:25.209479 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:46:25.209479 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:46:25.209479 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:46:25.209479 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:46:25.209479 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:46:25.209479 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:46:25.209479 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:46:25.209479 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:46:25.209479 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:46:25.209479 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:46:25.209479 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:46:25.209479 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:46:25.209479 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:46:25.209479 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:46:25.209479 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 09 00:46:25.210380 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 09 00:46:25.210380 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 09 00:46:25.210380 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:46:25.210380 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 649/1283] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:46:25 2022] GET /identity/v3/domains/Default => generated 87 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:46:25.215023 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e6194ab-3523-4854-a432-8255bbcde434 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:25.308889 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e6194ab-3523-4854-a432-8255bbcde434 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:25.341642 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4184e096-5963-446c-89eb-dff6bd11b878 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:25.342224 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4184e096-5963-446c-89eb-dff6bd11b878 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:25.342500 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4184e096-5963-446c-89eb-dff6bd11b878 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:25.342767 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4184e096-5963-446c-89eb-dff6bd11b878 None admin] PATH_INFO: `/v3/domains` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:25.343385 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4184e096-5963-446c-89eb-dff6bd11b878 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:46:25.343710 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4184e096-5963-446c-89eb-dff6bd11b878 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:46:25.345053 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4184e096-5963-446c-89eb-dff6bd11b878 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:46:25.355575 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 635/1284] 10.222.0.33 () {56 vars in 1109 bytes} [Sun Oct 9 00:46:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:46:25.360944 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d8dd188-3973-47cb-9a4e-96bf8ce88dd2 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:25.457025 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d8dd188-3973-47cb-9a4e-96bf8ce88dd2 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:25.470563 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:46:25.490312 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b54e8c4-9d70-41b7-a7ea-04555d2c81d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:25.490920 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b54e8c4-9d70-41b7-a7ea-04555d2c81d0 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:25.491201 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b54e8c4-9d70-41b7-a7ea-04555d2c81d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:25.491477 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b54e8c4-9d70-41b7-a7ea-04555d2c81d0 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:25.498408 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b54e8c4-9d70-41b7-a7ea-04555d2c81d0 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:46:25.499858 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b54e8c4-9d70-41b7-a7ea-04555d2c81d0 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:46:25.510696 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application [None req-8b54e8c4-9d70-41b7-a7ea-04555d2c81d0 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 09 00:46:25.510696 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:46:25.510696 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:46:25.510696 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:46:25.510696 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:46:25.510696 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:46:25.510696 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:46:25.510696 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:46:25.510696 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:46:25.510696 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:46:25.510696 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:46:25.510696 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:46:25.510696 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 09 00:46:25.510696 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 09 00:46:25.510696 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 09 00:46:25.511736 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 09 00:46:25.511736 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:46:25.511736 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:46:25.511736 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:46:25.511736 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:46:25.511736 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:46:25.511736 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:46:25.511736 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:46:25.511736 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application with Lock( Oct 09 00:46:25.511736 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:46:25.511736 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:46:25.511736 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:46:25.511736 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:46:25.511736 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:46:25.511736 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:46:25.511736 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:46:25.511736 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:46:25.512711 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 09 00:46:25.512711 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 09 00:46:25.512711 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 09 00:46:25.512711 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 09 00:46:25.512711 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 09 00:46:25.512711 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 09 00:46:25.512711 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 09 00:46:25.512711 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: ERROR keystone.server.flask.application Oct 09 00:46:25.512711 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 650/1285] 10.222.0.33 () {56 vars in 1111 bytes} [Sun Oct 9 00:46:25 2022] GET /identity/v3/projects/service => generated 88 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:46:25.517987 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4184e096-5963-446c-89eb-dff6bd11b878 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:25.565871 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:46:25.613950 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4184e096-5963-446c-89eb-dff6bd11b878 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:25.646452 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4418fa62-b12d-4cc2-8ccf-6f096c54672d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:25.647038 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4418fa62-b12d-4cc2-8ccf-6f096c54672d None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:25.647313 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4418fa62-b12d-4cc2-8ccf-6f096c54672d None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:25.647580 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4418fa62-b12d-4cc2-8ccf-6f096c54672d None admin] PATH_INFO: `/v3/projects` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:25.648392 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4418fa62-b12d-4cc2-8ccf-6f096c54672d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:46:25.648699 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4418fa62-b12d-4cc2-8ccf-6f096c54672d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:46:25.650098 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4418fa62-b12d-4cc2-8ccf-6f096c54672d None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:46:25.661443 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 636/1286] 10.222.0.33 () {56 vars in 1148 bytes} [Sun Oct 9 00:46:25 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:46:25.667222 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b54e8c4-9d70-41b7-a7ea-04555d2c81d0 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:25.765598 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b54e8c4-9d70-41b7-a7ea-04555d2c81d0 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:25.798769 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40d1564d-f489-4e26-9d26-1925c47e1ecf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:25.799353 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d1564d-f489-4e26-9d26-1925c47e1ecf None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:25.799657 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d1564d-f489-4e26-9d26-1925c47e1ecf None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:25.799933 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d1564d-f489-4e26-9d26-1925c47e1ecf None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:25.800642 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40d1564d-f489-4e26-9d26-1925c47e1ecf None admin] RBAC: Adding query filter params (role.id=092036ee46ac427baf4d56468575f697, scope.project.id=013c83476b1340fbb051061833c4a9c6, user.id=d25e240cb93643ed8122ccac9586936f) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:46:25.800953 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40d1564d-f489-4e26-9d26-1925c47e1ecf None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:46:25.802372 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40d1564d-f489-4e26-9d26-1925c47e1ecf None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:46:25.807997 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 651/1287] 10.222.0.33 () {56 vars in 1374 bytes} [Sun Oct 9 00:46:25 2022] GET /identity/v3/role_assignments?user.id=d25e240cb93643ed8122ccac9586936f&scope.project.id=013c83476b1340fbb051061833c4a9c6&role.id=092036ee46ac427baf4d56468575f697 => generated 608 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:46:27.946501 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c43b89c8-a75c-41e5-b672-9701eb8ea8cd None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:27.947032 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c43b89c8-a75c-41e5-b672-9701eb8ea8cd None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:27.947032 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c43b89c8-a75c-41e5-b672-9701eb8ea8cd None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:27.948388 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 637/1288] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:46:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:46:27.953446 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-078ec482-b286-41e5-8a09-f76080549a9a None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:27.953866 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-078ec482-b286-41e5-8a09-f76080549a9a None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:27.954000 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-078ec482-b286-41e5-8a09-f76080549a9a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:27.972249 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-078ec482-b286-41e5-8a09-f76080549a9a None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:46:27.974378 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-078ec482-b286-41e5-8a09-f76080549a9a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:46:28.003239 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:46:28.040452 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 652/1289] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:46:27 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 09 00:46:28.308568 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c43b89c8-a75c-41e5-b672-9701eb8ea8cd None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:28.404799 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c43b89c8-a75c-41e5-b672-9701eb8ea8cd None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:28.414846 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:46:28.438724 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a37d0d2-ee05-4a36-b324-5407e14d31a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:28.439342 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a37d0d2-ee05-4a36-b324-5407e14d31a8 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:28.439655 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a37d0d2-ee05-4a36-b324-5407e14d31a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:28.439934 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a37d0d2-ee05-4a36-b324-5407e14d31a8 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:28.440509 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a37d0d2-ee05-4a36-b324-5407e14d31a8 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:46:28.441862 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a37d0d2-ee05-4a36-b324-5407e14d31a8 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:46:28.450901 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-9a37d0d2-ee05-4a36-b324-5407e14d31a8 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Oct 09 00:46:28.450901 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:46:28.450901 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 09 00:46:28.450901 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 09 00:46:28.450901 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 09 00:46:28.450901 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 09 00:46:28.450901 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 09 00:46:28.450901 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 09 00:46:28.450901 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 09 00:46:28.450901 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:46:28.450901 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 09 00:46:28.450901 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:46:28.450901 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:46:28.450901 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:46:28.450901 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:46:28.450901 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:46:28.450901 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:46:28.452131 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:46:28.452131 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:46:28.452131 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:46:28.452131 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:46:28.452131 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:46:28.452131 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:46:28.452131 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 09 00:46:28.452131 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 09 00:46:28.452131 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 09 00:46:28.452131 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 09 00:46:28.452131 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:46:28.452131 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:46:28.452131 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:46:28.452131 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:46:28.452131 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:46:28.452131 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:46:28.452974 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:46:28.452974 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:46:28.452974 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:46:28.452974 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:46:28.452974 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:46:28.452974 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:46:28.452974 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:46:28.452974 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:46:28.452974 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:46:28.452974 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:46:28.452974 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 09 00:46:28.452974 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 09 00:46:28.452974 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 09 00:46:28.452974 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:46:28.452974 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 638/1290] 10.222.0.33 () {56 vars in 1117 bytes} [Sun Oct 9 00:46:28 2022] GET /identity/v3/services/placement => generated 90 bytes in 144 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:46:28.457802 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-078ec482-b286-41e5-8a09-f76080549a9a None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:28.553434 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-078ec482-b286-41e5-8a09-f76080549a9a None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:28.586582 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-718d1f00-2a0b-4611-a31e-a400493ccd8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:28.587143 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718d1f00-2a0b-4611-a31e-a400493ccd8f None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:28.587420 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718d1f00-2a0b-4611-a31e-a400493ccd8f None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:28.587719 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718d1f00-2a0b-4611-a31e-a400493ccd8f None admin] PATH_INFO: `/v3/services` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:28.588336 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-718d1f00-2a0b-4611-a31e-a400493ccd8f None admin] RBAC: Adding query filter params (name=placement) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:46:28.588634 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-718d1f00-2a0b-4611-a31e-a400493ccd8f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:46:28.589980 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-718d1f00-2a0b-4611-a31e-a400493ccd8f None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:46:28.594608 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 653/1291] 10.222.0.33 () {56 vars in 1116 bytes} [Sun Oct 9 00:46:28 2022] GET /identity/v3/services?name=placement => generated 126 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:46:28.600124 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a37d0d2-ee05-4a36-b324-5407e14d31a8 None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:28.695725 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a37d0d2-ee05-4a36-b324-5407e14d31a8 None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:28.727759 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73c3acd5-88ec-497d-8426-909031afa289 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:28.728344 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73c3acd5-88ec-497d-8426-909031afa289 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:28.728624 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73c3acd5-88ec-497d-8426-909031afa289 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:28.728892 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73c3acd5-88ec-497d-8426-909031afa289 None admin] PATH_INFO: `/v3/services` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:28.729501 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73c3acd5-88ec-497d-8426-909031afa289 None admin] RBAC: Adding query filter params (type=placement) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:46:28.729801 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73c3acd5-88ec-497d-8426-909031afa289 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:46:28.731135 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73c3acd5-88ec-497d-8426-909031afa289 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:46:28.735520 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 639/1292] 10.222.0.33 () {56 vars in 1116 bytes} [Sun Oct 9 00:46:28 2022] GET /identity/v3/services?type=placement => generated 126 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:46:30.857381 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60192d74-9d16-4bd2-84b1-07a6d751dfb7 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:30.857953 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60192d74-9d16-4bd2-84b1-07a6d751dfb7 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:30.857953 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60192d74-9d16-4bd2-84b1-07a6d751dfb7 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:30.859242 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 654/1293] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:46:30 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:46:30.864587 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6230c315-4d18-4ec7-9680-e7c81e4028fe None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:30.865007 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6230c315-4d18-4ec7-9680-e7c81e4028fe None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:30.865123 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6230c315-4d18-4ec7-9680-e7c81e4028fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:30.882984 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-6230c315-4d18-4ec7-9680-e7c81e4028fe None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:46:30.884202 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-6230c315-4d18-4ec7-9680-e7c81e4028fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:46:30.912872 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:46:30.949048 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 640/1294] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:46:30 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 09 00:46:31.222165 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60192d74-9d16-4bd2-84b1-07a6d751dfb7 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:31.252363 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:46:31.321150 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60192d74-9d16-4bd2-84b1-07a6d751dfb7 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:31.354409 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8dd0ccd-9a1a-4988-9082-9f8c690d5014 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:31.354996 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8dd0ccd-9a1a-4988-9082-9f8c690d5014 None admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:31.355276 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8dd0ccd-9a1a-4988-9082-9f8c690d5014 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:31.355547 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8dd0ccd-9a1a-4988-9082-9f8c690d5014 None admin] PATH_INFO: `/v3/services` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:31.356225 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8dd0ccd-9a1a-4988-9082-9f8c690d5014 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:46:31.357611 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8dd0ccd-9a1a-4988-9082-9f8c690d5014 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:46:31.370892 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 655/1295] 10.222.0.33 () {60 vars in 1141 bytes} [Sun Oct 9 00:46:31 2022] POST /identity/v3/services => generated 246 bytes in 149 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:46:33.495901 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb2b705-d423-4c87-bd2a-ed7b6b0b7d29 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:33.496423 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb2b705-d423-4c87-bd2a-ed7b6b0b7d29 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:33.496423 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb2b705-d423-4c87-bd2a-ed7b6b0b7d29 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:33.497770 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 641/1296] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:46:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:46:33.502956 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b18ee51-c52a-428f-b89a-d6966c2e4ebf None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:33.503368 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b18ee51-c52a-428f-b89a-d6966c2e4ebf None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:33.503492 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b18ee51-c52a-428f-b89a-d6966c2e4ebf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:33.521601 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-2b18ee51-c52a-428f-b89a-d6966c2e4ebf None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:46:33.522759 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-2b18ee51-c52a-428f-b89a-d6966c2e4ebf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:46:33.592824 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 656/1297] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:46:33 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 09 00:46:33.859278 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adb2b705-d423-4c87-bd2a-ed7b6b0b7d29 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:33.956865 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adb2b705-d423-4c87-bd2a-ed7b6b0b7d29 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:33.988743 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8364dffa-edf1-4b13-9362-1b3eb5ef17dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:33.989353 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8364dffa-edf1-4b13-9362-1b3eb5ef17dd None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:33.989637 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8364dffa-edf1-4b13-9362-1b3eb5ef17dd None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:33.989910 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8364dffa-edf1-4b13-9362-1b3eb5ef17dd None admin] PATH_INFO: `/v3/services/placement` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:33.990486 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8364dffa-edf1-4b13-9362-1b3eb5ef17dd None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:46:33.991856 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8364dffa-edf1-4b13-9362-1b3eb5ef17dd None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:46:34.000893 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-8364dffa-edf1-4b13-9362-1b3eb5ef17dd None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Oct 09 00:46:34.000893 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:46:34.000893 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 09 00:46:34.000893 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 09 00:46:34.000893 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 09 00:46:34.000893 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 09 00:46:34.000893 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 09 00:46:34.000893 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 09 00:46:34.000893 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 09 00:46:34.000893 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:46:34.000893 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 09 00:46:34.000893 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:46:34.000893 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:46:34.000893 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:46:34.000893 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:46:34.000893 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:46:34.000893 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:46:34.002013 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:46:34.002013 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:46:34.002013 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:46:34.002013 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:46:34.002013 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:46:34.002013 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:46:34.002013 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 09 00:46:34.002013 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 09 00:46:34.002013 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 09 00:46:34.002013 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 09 00:46:34.002013 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:46:34.002013 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:46:34.002013 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:46:34.002013 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:46:34.002013 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:46:34.002013 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:46:34.002875 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:46:34.002875 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:46:34.002875 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:46:34.002875 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:46:34.002875 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:46:34.002875 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:46:34.002875 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:46:34.002875 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:46:34.002875 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:46:34.002875 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:46:34.002875 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 09 00:46:34.002875 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 09 00:46:34.002875 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 09 00:46:34.002875 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:46:34.002875 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 642/1298] 10.222.0.33 () {56 vars in 1117 bytes} [Sun Oct 9 00:46:33 2022] GET /identity/v3/services/placement => generated 90 bytes in 143 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:46:34.007870 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b18ee51-c52a-428f-b89a-d6966c2e4ebf None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:34.026474 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:46:34.107417 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b18ee51-c52a-428f-b89a-d6966c2e4ebf None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:34.140560 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b13cacaf-3c50-488b-8287-8effddd8a0a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:34.141105 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b13cacaf-3c50-488b-8287-8effddd8a0a4 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:34.141383 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b13cacaf-3c50-488b-8287-8effddd8a0a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:34.141653 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b13cacaf-3c50-488b-8287-8effddd8a0a4 None admin] PATH_INFO: `/v3/services` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:34.142271 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b13cacaf-3c50-488b-8287-8effddd8a0a4 None admin] RBAC: Adding query filter params (name=placement) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:46:34.142568 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b13cacaf-3c50-488b-8287-8effddd8a0a4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:46:34.143945 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b13cacaf-3c50-488b-8287-8effddd8a0a4 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:46:34.148932 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 657/1299] 10.222.0.33 () {56 vars in 1116 bytes} [Sun Oct 9 00:46:34 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:46:34.154869 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8364dffa-edf1-4b13-9362-1b3eb5ef17dd None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:34.186394 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:46:34.251735 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8364dffa-edf1-4b13-9362-1b3eb5ef17dd None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:34.282998 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93d27d6c-5ded-4ac8-84bf-0e43ae8ab9e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:34.283614 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d27d6c-5ded-4ac8-84bf-0e43ae8ab9e0 None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:34.283918 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d27d6c-5ded-4ac8-84bf-0e43ae8ab9e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:34.284194 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d27d6c-5ded-4ac8-84bf-0e43ae8ab9e0 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:34.284883 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93d27d6c-5ded-4ac8-84bf-0e43ae8ab9e0 None admin] RBAC: Adding query filter params (interface=public, service_id=c6c37c8583284163b6d9343a76f0d1d6, region_id=RegionOne) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:46:34.285190 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93d27d6c-5ded-4ac8-84bf-0e43ae8ab9e0 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:46:34.286531 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93d27d6c-5ded-4ac8-84bf-0e43ae8ab9e0 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:46:34.291328 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 643/1300] 10.222.0.33 () {56 vars in 1251 bytes} [Sun Oct 9 00:46:34 2022] GET /identity/v3/endpoints?service_id=c6c37c8583284163b6d9343a76f0d1d6&interface=public®ion_id=RegionOne => generated 194 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:46:34.297409 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b13cacaf-3c50-488b-8287-8effddd8a0a4 None admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:34.376324 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:46:34.393997 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b13cacaf-3c50-488b-8287-8effddd8a0a4 None admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:34.426920 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-addf89c2-c7bf-4aec-a043-029886634959 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:34.427504 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-addf89c2-c7bf-4aec-a043-029886634959 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:34.427810 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-addf89c2-c7bf-4aec-a043-029886634959 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:34.428083 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-addf89c2-c7bf-4aec-a043-029886634959 None admin] PATH_INFO: `/v3/services` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:34.428670 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-addf89c2-c7bf-4aec-a043-029886634959 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:46:34.430013 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-addf89c2-c7bf-4aec-a043-029886634959 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:46:34.437160 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 658/1301] 10.222.0.33 () {56 vars in 1087 bytes} [Sun Oct 9 00:46:34 2022] GET /identity/v3/services => generated 2191 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:46:36.543279 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20eeb01b-859f-4ba4-ba50-36fbd40822f7 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:36.543839 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20eeb01b-859f-4ba4-ba50-36fbd40822f7 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:36.543930 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20eeb01b-859f-4ba4-ba50-36fbd40822f7 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:36.545192 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 644/1302] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:46:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:46:36.550725 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c1856f8-0151-4b5c-8d5f-d33391fdc8c0 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:36.551141 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c1856f8-0151-4b5c-8d5f-d33391fdc8c0 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:36.551272 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c1856f8-0151-4b5c-8d5f-d33391fdc8c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:36.569658 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-0c1856f8-0151-4b5c-8d5f-d33391fdc8c0 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:46:36.570836 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-0c1856f8-0151-4b5c-8d5f-d33391fdc8c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:46:36.635923 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 659/1303] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:46:36 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 09 00:46:36.907383 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20eeb01b-859f-4ba4-ba50-36fbd40822f7 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:36.999554 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:46:37.006256 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20eeb01b-859f-4ba4-ba50-36fbd40822f7 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:37.038194 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-991689c5-d5b9-4085-b290-e22d7d7808dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:37.038803 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-991689c5-d5b9-4085-b290-e22d7d7808dc None admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:37.039083 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-991689c5-d5b9-4085-b290-e22d7d7808dc None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:37.039357 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-991689c5-d5b9-4085-b290-e22d7d7808dc None admin] PATH_INFO: `/v3/services/placement` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:37.039964 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-991689c5-d5b9-4085-b290-e22d7d7808dc None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:46:37.041307 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-991689c5-d5b9-4085-b290-e22d7d7808dc None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:46:37.050603 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application [None req-991689c5-d5b9-4085-b290-e22d7d7808dc None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Oct 09 00:46:37.050603 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:46:37.050603 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 09 00:46:37.050603 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 09 00:46:37.050603 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 09 00:46:37.050603 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 09 00:46:37.050603 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 09 00:46:37.050603 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 09 00:46:37.050603 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 09 00:46:37.050603 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:46:37.050603 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 09 00:46:37.050603 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:46:37.050603 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 09 00:46:37.050603 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 09 00:46:37.050603 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 09 00:46:37.050603 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 09 00:46:37.050603 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 09 00:46:37.051755 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 09 00:46:37.051755 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 09 00:46:37.051755 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 09 00:46:37.051755 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 09 00:46:37.051755 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 09 00:46:37.051755 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 09 00:46:37.051755 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 09 00:46:37.051755 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 09 00:46:37.051755 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 09 00:46:37.051755 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 09 00:46:37.051755 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 09 00:46:37.051755 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 09 00:46:37.051755 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 09 00:46:37.051755 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 09 00:46:37.051755 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 09 00:46:37.051755 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.get_or_create( Oct 09 00:46:37.052964 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 09 00:46:37.052964 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application with Lock( Oct 09 00:46:37.052964 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 09 00:46:37.052964 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self._enter() Oct 09 00:46:37.052964 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 09 00:46:37.052964 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 09 00:46:37.052964 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 09 00:46:37.052964 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application return self.creator() Oct 09 00:46:37.052964 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 09 00:46:37.052964 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application created_value = creator( Oct 09 00:46:37.052964 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 09 00:46:37.052964 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 09 00:46:37.052964 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 09 00:46:37.052964 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: ERROR keystone.server.flask.application Oct 09 00:46:37.052964 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 645/1304] 10.222.0.33 () {56 vars in 1117 bytes} [Sun Oct 9 00:46:36 2022] GET /identity/v3/services/placement => generated 90 bytes in 145 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 09 00:46:37.057477 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c1856f8-0151-4b5c-8d5f-d33391fdc8c0 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:37.131263 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:46:37.156181 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c1856f8-0151-4b5c-8d5f-d33391fdc8c0 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:37.188490 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a34eebcf-525d-4cc4-bbc2-75999c237667 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:37.189078 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a34eebcf-525d-4cc4-bbc2-75999c237667 None admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:37.189351 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a34eebcf-525d-4cc4-bbc2-75999c237667 None admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:37.189616 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a34eebcf-525d-4cc4-bbc2-75999c237667 None admin] PATH_INFO: `/v3/services` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:37.190229 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a34eebcf-525d-4cc4-bbc2-75999c237667 None admin] RBAC: Adding query filter params (name=placement) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:46:37.190523 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a34eebcf-525d-4cc4-bbc2-75999c237667 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:46:37.191890 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a34eebcf-525d-4cc4-bbc2-75999c237667 None admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:46:37.196803 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 660/1305] 10.222.0.33 () {56 vars in 1116 bytes} [Sun Oct 9 00:46:37 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:46:37.203008 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-991689c5-d5b9-4085-b290-e22d7d7808dc None admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:46:37.299083 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-991689c5-d5b9-4085-b290-e22d7d7808dc None admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:46:37.331213 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-827a1d73-b752-49fb-9070-b0fcb936ee79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:46:37.331829 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-827a1d73-b752-49fb-9070-b0fcb936ee79 None admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:46:37.332107 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-827a1d73-b752-49fb-9070-b0fcb936ee79 None admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:46:37.332376 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-827a1d73-b752-49fb-9070-b0fcb936ee79 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:46:37.333030 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-827a1d73-b752-49fb-9070-b0fcb936ee79 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:46:37.334367 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-827a1d73-b752-49fb-9070-b0fcb936ee79 None admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:46:37.354121 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 646/1306] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:46:37 2022] POST /identity/v3/endpoints => generated 331 bytes in 152 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:46:47.193925 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=30 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:46:47.351135 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=91,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:47:40.772459 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7f2b9e9-b7b6-41f1-b692-0d9eefc22f81 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:47:40.773387 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7f2b9e9-b7b6-41f1-b692-0d9eefc22f81 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:47:40.773807 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7f2b9e9-b7b6-41f1-b692-0d9eefc22f81 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:47:40.775275 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 661/1307] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:47:40 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:47:40.780534 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3554baea-9110-4849-9bfa-d51d25a2d7eb None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:47:40.780974 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3554baea-9110-4849-9bfa-d51d25a2d7eb None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:47:40.781369 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3554baea-9110-4849-9bfa-d51d25a2d7eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:47:40.956160 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-3554baea-9110-4849-9bfa-d51d25a2d7eb None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:47:40.957375 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-3554baea-9110-4849-9bfa-d51d25a2d7eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:47:41.163555 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 647/1308] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:47:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 384 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:47:41.471728 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14106ca5-44d1-44e0-8627-efdd62365300 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:47:41.472634 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14106ca5-44d1-44e0-8627-efdd62365300 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:47:41.473112 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14106ca5-44d1-44e0-8627-efdd62365300 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:47:41.475364 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 662/1309] 10.222.0.33 () {54 vars in 964 bytes} [Sun Oct 9 00:47:41 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:47:41.481438 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c64bed7a-5be2-4433-87b5-f9a657897ecd None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:47:41.481756 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c64bed7a-5be2-4433-87b5-f9a657897ecd None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:47:41.482001 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c64bed7a-5be2-4433-87b5-f9a657897ecd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:47:41.573976 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-c64bed7a-5be2-4433-87b5-f9a657897ecd None None] MFA Rules not processed for user `7d2eabf9865847ceb6e44d55126078a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:47:41.575269 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-c64bed7a-5be2-4433-87b5-f9a657897ecd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:47:41.600013 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:47:41.672215 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 648/1310] 10.222.0.33 () {58 vars in 1061 bytes} [Sun Oct 9 00:47:41 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:47:41.682649 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14106ca5-44d1-44e0-8627-efdd62365300 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:47:41.684805 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-14106ca5-44d1-44e0-8627-efdd62365300 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:47:41.816608 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14106ca5-44d1-44e0-8627-efdd62365300 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:47:41.902009 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-886afe01-e50c-441c-b5be-b43461b5c445 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:47:41.903007 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-886afe01-e50c-441c-b5be-b43461b5c445 service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:47:41.903514 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-886afe01-e50c-441c-b5be-b43461b5c445 service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:47:41.904149 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-886afe01-e50c-441c-b5be-b43461b5c445 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:47:41.905972 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-886afe01-e50c-441c-b5be-b43461b5c445 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:47:41.964971 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:47:42.024215 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-886afe01-e50c-441c-b5be-b43461b5c445 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:47:42.025994 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-886afe01-e50c-441c-b5be-b43461b5c445 service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:47:42.059939 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 663/1311] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:47:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 378 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:47:44.509801 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b6491e2-811e-4e33-9e5d-05ea3044d449 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:47:44.510267 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b6491e2-811e-4e33-9e5d-05ea3044d449 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:47:44.510358 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b6491e2-811e-4e33-9e5d-05ea3044d449 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:47:44.511683 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 649/1312] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:47:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:47:44.517382 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21d94ddd-1bde-43db-90f7-f6dab9fae43d None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:47:44.517784 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21d94ddd-1bde-43db-90f7-f6dab9fae43d None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:47:44.517933 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21d94ddd-1bde-43db-90f7-f6dab9fae43d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:47:44.589513 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-21d94ddd-1bde-43db-90f7-f6dab9fae43d None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:47:44.590718 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-21d94ddd-1bde-43db-90f7-f6dab9fae43d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:47:44.683572 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 664/1313] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:47:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:47:44.965075 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b6491e2-811e-4e33-9e5d-05ea3044d449 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:47:44.966636 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-1b6491e2-811e-4e33-9e5d-05ea3044d449 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:47:45.067286 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b6491e2-811e-4e33-9e5d-05ea3044d449 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:47:45.138725 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:47:45.140815 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4f06b1f-ca50-466c-b24d-5350fde4cee1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:47:45.141514 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4f06b1f-ca50-466c-b24d-5350fde4cee1 service neutron] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:47:45.142015 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4f06b1f-ca50-466c-b24d-5350fde4cee1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:47:45.142343 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4f06b1f-ca50-466c-b24d-5350fde4cee1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:47:45.144283 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-a4f06b1f-ca50-466c-b24d-5350fde4cee1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:47:45.251998 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4f06b1f-ca50-466c-b24d-5350fde4cee1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:47:45.253392 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4f06b1f-ca50-466c-b24d-5350fde4cee1 service neutron] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:47:45.271762 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 650/1314] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:47:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:47:47.677255 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09045ab9-81ac-463c-90b2-23b6fcec8036 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:47:47.677834 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09045ab9-81ac-463c-90b2-23b6fcec8036 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:47:47.677834 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09045ab9-81ac-463c-90b2-23b6fcec8036 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:47:47.679388 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 665/1315] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:47:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:47:47.684837 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3832a462-774d-42c2-9c64-535eac5165a5 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:47:47.685266 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3832a462-774d-42c2-9c64-535eac5165a5 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:47:47.685370 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3832a462-774d-42c2-9c64-535eac5165a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:47:47.749294 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-3832a462-774d-42c2-9c64-535eac5165a5 None None] MFA Rules not processed for user `da5fb3b5405648df8a2114ff2dca2d02`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:47:47.750492 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-3832a462-774d-42c2-9c64-535eac5165a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:47:47.802544 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:47:47.847714 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 651/1316] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:47:47 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:47:48.128184 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09045ab9-81ac-463c-90b2-23b6fcec8036 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:47:48.129711 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-09045ab9-81ac-463c-90b2-23b6fcec8036 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:47:48.139319 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:47:48.230570 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09045ab9-81ac-463c-90b2-23b6fcec8036 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:47:48.302204 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5364d126-a5e7-4790-a47c-426bc3209107 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:47:48.302796 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5364d126-a5e7-4790-a47c-426bc3209107 service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:47:48.303068 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5364d126-a5e7-4790-a47c-426bc3209107 service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:47:48.303335 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5364d126-a5e7-4790-a47c-426bc3209107 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:47:48.305184 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-5364d126-a5e7-4790-a47c-426bc3209107 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:47:48.366620 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:47:48.412606 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5364d126-a5e7-4790-a47c-426bc3209107 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:47:48.414042 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5364d126-a5e7-4790-a47c-426bc3209107 service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:47:48.432871 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 666/1317] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:47:48 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:47:51.181043 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2770033d-7987-4ce7-a105-280273ac8f79 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:47:51.181598 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2770033d-7987-4ce7-a105-280273ac8f79 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:47:51.181598 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2770033d-7987-4ce7-a105-280273ac8f79 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:47:51.182907 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 652/1318] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:47:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:47:51.188024 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7adee4c-78f0-44c3-89e1-481eb91cd7f5 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:47:51.188472 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7adee4c-78f0-44c3-89e1-481eb91cd7f5 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:47:51.188604 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7adee4c-78f0-44c3-89e1-481eb91cd7f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:47:51.256555 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-a7adee4c-78f0-44c3-89e1-481eb91cd7f5 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:47:51.257729 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-a7adee4c-78f0-44c3-89e1-481eb91cd7f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:47:51.351472 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 667/1319] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:47:51 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 09 00:47:51.632066 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2770033d-7987-4ce7-a105-280273ac8f79 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:47:51.633645 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-2770033d-7987-4ce7-a105-280273ac8f79 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:47:51.733317 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2770033d-7987-4ce7-a105-280273ac8f79 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:47:51.808324 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aee9bcd8-750b-4427-9467-7d3de23989bd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:47:51.808923 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee9bcd8-750b-4427-9467-7d3de23989bd service neutron] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:47:51.809205 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee9bcd8-750b-4427-9467-7d3de23989bd service neutron] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:47:51.809477 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee9bcd8-750b-4427-9467-7d3de23989bd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:47:51.811058 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-aee9bcd8-750b-4427-9467-7d3de23989bd service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:47:51.817879 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:47:51.915634 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aee9bcd8-750b-4427-9467-7d3de23989bd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:47:51.917053 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aee9bcd8-750b-4427-9467-7d3de23989bd service neutron] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:47:51.934983 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 653/1320] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:47:51 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 303 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:47:54.700664 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd7f57f-6992-4893-a958-3e0b3924db0f None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:47:54.701201 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd7f57f-6992-4893-a958-3e0b3924db0f None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:47:54.701201 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd7f57f-6992-4893-a958-3e0b3924db0f None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:47:54.702464 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 668/1321] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:47:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:47:54.708060 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fccc423-c111-43cd-8637-849b116f0a6a None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:47:54.708485 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fccc423-c111-43cd-8637-849b116f0a6a None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:47:54.708755 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fccc423-c111-43cd-8637-849b116f0a6a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:47:54.783672 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-3fccc423-c111-43cd-8637-849b116f0a6a None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:47:54.785385 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-3fccc423-c111-43cd-8637-849b116f0a6a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:47:54.849062 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:47:54.893601 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 654/1322] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:47:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:47:55.181684 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfd7f57f-6992-4893-a958-3e0b3924db0f None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:47:55.183209 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-cfd7f57f-6992-4893-a958-3e0b3924db0f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:47:55.202543 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:47:55.293390 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfd7f57f-6992-4893-a958-3e0b3924db0f None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:47:55.368426 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cf981ab-34c2-4046-9c4d-f6d8b093662e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:47:55.369015 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cf981ab-34c2-4046-9c4d-f6d8b093662e service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:47:55.369295 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cf981ab-34c2-4046-9c4d-f6d8b093662e service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:47:55.369567 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cf981ab-34c2-4046-9c4d-f6d8b093662e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:47:55.371205 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-0cf981ab-34c2-4046-9c4d-f6d8b093662e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:47:55.441640 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:47:55.481780 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cf981ab-34c2-4046-9c4d-f6d8b093662e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:47:55.483232 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cf981ab-34c2-4046-9c4d-f6d8b093662e service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:47:55.502287 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 669/1323] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:47:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 321 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:47:57.717169 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa39ce23-03c9-40af-ac9f-77f8f92191a5 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:47:57.717726 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa39ce23-03c9-40af-ac9f-77f8f92191a5 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:47:57.717726 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa39ce23-03c9-40af-ac9f-77f8f92191a5 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:47:57.719026 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 655/1324] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:47:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:47:57.724733 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d918ca-09ec-44ef-9a78-929290829724 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:47:57.725162 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d918ca-09ec-44ef-9a78-929290829724 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:47:57.725283 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d918ca-09ec-44ef-9a78-929290829724 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:47:57.800902 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-a4d918ca-09ec-44ef-9a78-929290829724 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:47:57.802117 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-a4d918ca-09ec-44ef-9a78-929290829724 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:47:57.902301 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 670/1325] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:47:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:47:58.185592 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa39ce23-03c9-40af-ac9f-77f8f92191a5 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:47:58.187156 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-aa39ce23-03c9-40af-ac9f-77f8f92191a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:47:58.289710 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa39ce23-03c9-40af-ac9f-77f8f92191a5 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:47:58.362028 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93443587-f0ce-42ff-8d28-c699ab22d66b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:47:58.362555 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93443587-f0ce-42ff-8d28-c699ab22d66b service neutron] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:47:58.362834 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93443587-f0ce-42ff-8d28-c699ab22d66b service neutron] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:47:58.363106 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93443587-f0ce-42ff-8d28-c699ab22d66b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:47:58.364730 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-93443587-f0ce-42ff-8d28-c699ab22d66b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:47:58.381516 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:47:58.474528 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93443587-f0ce-42ff-8d28-c699ab22d66b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:47:58.475999 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93443587-f0ce-42ff-8d28-c699ab22d66b service neutron] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:47:58.493483 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 656/1326] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:47:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:48:00.705300 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06182e4e-fa85-455a-930f-607e218e5b4e None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:48:00.705854 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06182e4e-fa85-455a-930f-607e218e5b4e None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:48:00.705854 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06182e4e-fa85-455a-930f-607e218e5b4e None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:48:00.707184 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 671/1327] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:48:00 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:48:00.713247 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-775093db-f98d-4c0e-845a-b7f23c525919 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:48:00.713670 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-775093db-f98d-4c0e-845a-b7f23c525919 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:48:00.713670 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-775093db-f98d-4c0e-845a-b7f23c525919 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:48:00.787949 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-775093db-f98d-4c0e-845a-b7f23c525919 None None] MFA Rules not processed for user `da5fb3b5405648df8a2114ff2dca2d02`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:48:00.789190 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-775093db-f98d-4c0e-845a-b7f23c525919 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:48:00.857065 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:48:00.889384 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 657/1328] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:48:00 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:48:01.172502 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06182e4e-fa85-455a-930f-607e218e5b4e None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:48:01.174060 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-06182e4e-fa85-455a-930f-607e218e5b4e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:48:01.206691 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:48:01.281341 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06182e4e-fa85-455a-930f-607e218e5b4e None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:48:01.357388 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc497985-7124-4151-8605-de954c4ae728 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:48:01.357986 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc497985-7124-4151-8605-de954c4ae728 service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:48:01.358263 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc497985-7124-4151-8605-de954c4ae728 service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:48:01.358533 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc497985-7124-4151-8605-de954c4ae728 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:48:01.360144 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-dc497985-7124-4151-8605-de954c4ae728 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:48:01.436628 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:48:01.470018 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc497985-7124-4151-8605-de954c4ae728 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:48:01.472075 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc497985-7124-4151-8605-de954c4ae728 service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:48:01.489912 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 672/1329] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:48:01 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 318 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:48:03.844342 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6c08861-63f7-447f-84d2-69649bcd8303 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:48:03.844951 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6c08861-63f7-447f-84d2-69649bcd8303 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:48:03.844951 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6c08861-63f7-447f-84d2-69649bcd8303 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:48:03.846250 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 658/1330] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:48:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:48:03.851445 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7340b1d3-3054-4c32-9d3a-49e707203603 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:48:03.851899 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7340b1d3-3054-4c32-9d3a-49e707203603 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:48:03.852037 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7340b1d3-3054-4c32-9d3a-49e707203603 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:48:03.915923 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-7340b1d3-3054-4c32-9d3a-49e707203603 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:48:03.917104 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-7340b1d3-3054-4c32-9d3a-49e707203603 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:48:04.011102 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 673/1331] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:48:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:48:04.289749 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6c08861-63f7-447f-84d2-69649bcd8303 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:48:04.291289 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-b6c08861-63f7-447f-84d2-69649bcd8303 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:48:04.392860 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6c08861-63f7-447f-84d2-69649bcd8303 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:48:04.464550 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae608b28-17e0-4ff0-ba2b-931a273a47f7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:48:04.465152 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae608b28-17e0-4ff0-ba2b-931a273a47f7 service neutron] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:48:04.465455 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae608b28-17e0-4ff0-ba2b-931a273a47f7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:48:04.465732 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae608b28-17e0-4ff0-ba2b-931a273a47f7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:48:04.467295 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-ae608b28-17e0-4ff0-ba2b-931a273a47f7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:48:04.494575 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:48:04.575575 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae608b28-17e0-4ff0-ba2b-931a273a47f7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:48:04.576972 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae608b28-17e0-4ff0-ba2b-931a273a47f7 service neutron] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:48:04.594485 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 659/1332] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:48:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:48:06.779276 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d16ab0-d48d-4447-b070-b76048c3fc7b None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:48:06.779810 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d16ab0-d48d-4447-b070-b76048c3fc7b None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:48:06.779810 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d16ab0-d48d-4447-b070-b76048c3fc7b None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:48:06.781113 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 674/1333] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:48:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:48:06.786590 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6409aca6-847b-47e0-ad52-e645f97bf278 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:48:06.787015 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6409aca6-847b-47e0-ad52-e645f97bf278 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:48:06.787141 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6409aca6-847b-47e0-ad52-e645f97bf278 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:48:06.853375 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-6409aca6-847b-47e0-ad52-e645f97bf278 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:48:06.854572 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-6409aca6-847b-47e0-ad52-e645f97bf278 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:48:06.926302 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:48:06.948569 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 660/1334] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:48:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:48:07.248360 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe0cc45e-a74d-49ee-84fb-a5350c68f3f2 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:48:07.248642 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe0cc45e-a74d-49ee-84fb-a5350c68f3f2 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:48:07.248916 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe0cc45e-a74d-49ee-84fb-a5350c68f3f2 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:48:07.250259 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 675/1335] 10.222.0.33 () {54 vars in 964 bytes} [Sun Oct 9 00:48:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 09 00:48:07.256648 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd27401-5087-4500-9417-0e1e3f9d92f1 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:48:07.256936 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd27401-5087-4500-9417-0e1e3f9d92f1 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:48:07.257213 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd27401-5087-4500-9417-0e1e3f9d92f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:48:07.334603 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-bbd27401-5087-4500-9417-0e1e3f9d92f1 None None] MFA Rules not processed for user `7d2eabf9865847ceb6e44d55126078a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:48:07.335748 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-bbd27401-5087-4500-9417-0e1e3f9d92f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:48:07.408488 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 661/1336] 10.222.0.33 () {58 vars in 1061 bytes} [Sun Oct 9 00:48:07 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:48:07.417327 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe0cc45e-a74d-49ee-84fb-a5350c68f3f2 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:48:07.418853 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-fe0cc45e-a74d-49ee-84fb-a5350c68f3f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:48:07.458472 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:48:07.523889 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe0cc45e-a74d-49ee-84fb-a5350c68f3f2 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:48:07.597669 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d9eb96-b8ba-4904-ba5f-1b731a771baa service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:48:07.598197 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d9eb96-b8ba-4904-ba5f-1b731a771baa service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:48:07.598475 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d9eb96-b8ba-4904-ba5f-1b731a771baa service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:48:07.598748 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d9eb96-b8ba-4904-ba5f-1b731a771baa service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:48:07.600378 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-19d9eb96-b8ba-4904-ba5f-1b731a771baa service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:48:07.682365 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:48:07.708710 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19d9eb96-b8ba-4904-ba5f-1b731a771baa service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:48:07.710142 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19d9eb96-b8ba-4904-ba5f-1b731a771baa service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:48:07.727669 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 676/1337] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:48:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:48:10.671110 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59d27221-7364-432d-9fba-66deee9ace2b None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:48:10.671666 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59d27221-7364-432d-9fba-66deee9ace2b None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:48:10.671666 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59d27221-7364-432d-9fba-66deee9ace2b None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:48:10.672994 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 662/1338] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:48:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:48:10.678584 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3474680-1a51-4d4f-a71e-5af74a73e919 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:48:10.678979 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3474680-1a51-4d4f-a71e-5af74a73e919 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:48:10.679128 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3474680-1a51-4d4f-a71e-5af74a73e919 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:48:10.744122 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-b3474680-1a51-4d4f-a71e-5af74a73e919 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:48:10.745239 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-b3474680-1a51-4d4f-a71e-5af74a73e919 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:48:10.842787 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 677/1339] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:48:10 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:48:11.122709 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59d27221-7364-432d-9fba-66deee9ace2b None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:48:11.124276 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-59d27221-7364-432d-9fba-66deee9ace2b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:48:11.198584 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:48:11.226781 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59d27221-7364-432d-9fba-66deee9ace2b None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:48:11.299542 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae9ba638-410f-49b7-a31f-8b795493b4c4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:48:11.300174 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae9ba638-410f-49b7-a31f-8b795493b4c4 service neutron] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:48:11.300444 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae9ba638-410f-49b7-a31f-8b795493b4c4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:48:11.300707 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae9ba638-410f-49b7-a31f-8b795493b4c4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:48:11.302305 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-ae9ba638-410f-49b7-a31f-8b795493b4c4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:48:11.410928 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae9ba638-410f-49b7-a31f-8b795493b4c4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:48:11.412390 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae9ba638-410f-49b7-a31f-8b795493b4c4 service neutron] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:48:11.429720 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 663/1340] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:48:11 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:48:15.628783 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cba7f6f-6443-4e44-9b52-69ff05c583c5 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:48:15.629309 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cba7f6f-6443-4e44-9b52-69ff05c583c5 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:48:15.629309 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cba7f6f-6443-4e44-9b52-69ff05c583c5 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:48:15.630828 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 678/1341] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:48:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:48:15.635900 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b7beed-6212-43a9-b868-dddc9b66f491 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:48:15.636291 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b7beed-6212-43a9-b868-dddc9b66f491 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:48:15.636573 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b7beed-6212-43a9-b868-dddc9b66f491 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:48:15.642291 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:48:15.707402 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-52b7beed-6212-43a9-b868-dddc9b66f491 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:48:15.708564 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-52b7beed-6212-43a9-b868-dddc9b66f491 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:48:15.807461 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 664/1342] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:48:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:48:16.089144 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cba7f6f-6443-4e44-9b52-69ff05c583c5 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:48:16.090741 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-6cba7f6f-6443-4e44-9b52-69ff05c583c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:48:16.142015 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:48:16.199935 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cba7f6f-6443-4e44-9b52-69ff05c583c5 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:48:16.274055 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b10c1b4b-b800-400e-8721-e1a9bf11f11b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:48:16.274662 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10c1b4b-b800-400e-8721-e1a9bf11f11b service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:48:16.274938 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10c1b4b-b800-400e-8721-e1a9bf11f11b service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:48:16.275204 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10c1b4b-b800-400e-8721-e1a9bf11f11b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:48:16.276826 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-b10c1b4b-b800-400e-8721-e1a9bf11f11b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:48:16.371559 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:48:16.388770 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b10c1b4b-b800-400e-8721-e1a9bf11f11b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:48:16.390202 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b10c1b4b-b800-400e-8721-e1a9bf11f11b service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:48:16.408098 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 679/1343] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:48:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 319 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:48:19.164999 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b4f3412-0178-4828-bd2a-f777044594ad None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:48:19.165529 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b4f3412-0178-4828-bd2a-f777044594ad None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:48:19.165529 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b4f3412-0178-4828-bd2a-f777044594ad None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:48:19.166852 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 665/1344] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:48:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:48:19.171986 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf24e9b4-ff45-4438-8d1a-262d83e7b684 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:48:19.172424 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf24e9b4-ff45-4438-8d1a-262d83e7b684 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:48:19.172552 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf24e9b4-ff45-4438-8d1a-262d83e7b684 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:48:19.238277 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-cf24e9b4-ff45-4438-8d1a-262d83e7b684 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:48:19.239450 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-cf24e9b4-ff45-4438-8d1a-262d83e7b684 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:48:19.336265 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 680/1345] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:48:19 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:48:19.619021 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b4f3412-0178-4828-bd2a-f777044594ad None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:48:19.620652 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-8b4f3412-0178-4828-bd2a-f777044594ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:48:19.709934 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:48:19.729518 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b4f3412-0178-4828-bd2a-f777044594ad None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:48:19.808903 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86bb3467-f36f-42e3-add7-261a3aea17cf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:48:19.809550 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86bb3467-f36f-42e3-add7-261a3aea17cf service neutron] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:48:19.809829 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86bb3467-f36f-42e3-add7-261a3aea17cf service neutron] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:48:19.810256 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86bb3467-f36f-42e3-add7-261a3aea17cf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:48:19.812209 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-86bb3467-f36f-42e3-add7-261a3aea17cf service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:48:19.921682 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86bb3467-f36f-42e3-add7-261a3aea17cf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:48:19.923270 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86bb3467-f36f-42e3-add7-261a3aea17cf service neutron] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:48:19.942990 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 666/1346] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:48:19 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 325 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:48:24.023697 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b2d5efc-5bf4-4659-9a2f-fd10d3192e33 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:48:24.024182 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b2d5efc-5bf4-4659-9a2f-fd10d3192e33 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:48:24.024299 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b2d5efc-5bf4-4659-9a2f-fd10d3192e33 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:48:24.025590 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 681/1347] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:48:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:48:24.030743 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72c67b78-62bb-419e-a0f6-a291deeba755 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:48:24.031167 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72c67b78-62bb-419e-a0f6-a291deeba755 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:48:24.031292 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72c67b78-62bb-419e-a0f6-a291deeba755 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:48:24.047160 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:48:24.100177 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-72c67b78-62bb-419e-a0f6-a291deeba755 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:48:24.101486 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-72c67b78-62bb-419e-a0f6-a291deeba755 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:48:24.202653 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 667/1348] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:48:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 09 00:48:24.482562 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b2d5efc-5bf4-4659-9a2f-fd10d3192e33 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:48:24.484238 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-5b2d5efc-5bf4-4659-9a2f-fd10d3192e33 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:48:24.544608 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:48:24.592150 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b2d5efc-5bf4-4659-9a2f-fd10d3192e33 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:48:24.666190 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3019ce05-4ced-44f1-b972-d4d8c8889fbb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:48:24.666790 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3019ce05-4ced-44f1-b972-d4d8c8889fbb service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:48:24.667099 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3019ce05-4ced-44f1-b972-d4d8c8889fbb service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:48:24.667373 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3019ce05-4ced-44f1-b972-d4d8c8889fbb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:48:24.669011 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-3019ce05-4ced-44f1-b972-d4d8c8889fbb service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:48:24.780470 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:48:24.785691 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3019ce05-4ced-44f1-b972-d4d8c8889fbb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:48:24.787428 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3019ce05-4ced-44f1-b972-d4d8c8889fbb service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:48:24.806132 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 682/1349] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:48:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 324 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:48:29.675328 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b738fc-9ebc-4ef8-9cc7-6b7ef6473c53 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:48:29.676204 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b738fc-9ebc-4ef8-9cc7-6b7ef6473c53 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:48:29.676636 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b738fc-9ebc-4ef8-9cc7-6b7ef6473c53 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:48:29.678396 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 668/1350] 10.222.0.33 () {54 vars in 921 bytes} [Sun Oct 9 00:48:29 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:48:29.685066 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25009b41-569e-41c5-b2c7-c1b4335f7ab6 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:48:29.685504 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25009b41-569e-41c5-b2c7-c1b4335f7ab6 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:48:29.685598 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25009b41-569e-41c5-b2c7-c1b4335f7ab6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:48:29.815317 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-25009b41-569e-41c5-b2c7-c1b4335f7ab6 None None] MFA Rules not processed for user `d25e240cb93643ed8122ccac9586936f`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:48:29.815317 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-25009b41-569e-41c5-b2c7-c1b4335f7ab6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:48:29.968348 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 683/1351] 10.222.0.33 () {58 vars in 1018 bytes} [Sun Oct 9 00:48:29 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 284 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:48:34.090742 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7895a3d5-d5e3-4d1a-a1a0-14dcee8f38c6 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:48:34.091185 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7895a3d5-d5e3-4d1a-a1a0-14dcee8f38c6 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:48:34.091294 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7895a3d5-d5e3-4d1a-a1a0-14dcee8f38c6 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:48:34.092800 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 669/1352] 10.222.0.33 () {54 vars in 921 bytes} [Sun Oct 9 00:48:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:48:34.097719 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38a1ac14-be12-4186-9c3f-6bb60befe877 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:48:34.098756 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38a1ac14-be12-4186-9c3f-6bb60befe877 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:48:34.099238 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38a1ac14-be12-4186-9c3f-6bb60befe877 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:48:34.161891 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:48:34.191650 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-38a1ac14-be12-4186-9c3f-6bb60befe877 None None] MFA Rules not processed for user `d25e240cb93643ed8122ccac9586936f`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:48:34.193318 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-38a1ac14-be12-4186-9c3f-6bb60befe877 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:48:34.201206 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=59 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:48:34.313734 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 684/1353] 10.222.0.33 () {58 vars in 1018 bytes} [Sun Oct 9 00:48:34 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 217 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:48:35.580440 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a7d5636-3bc1-48b2-ab59-435f3f963176 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:48:35.580991 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a7d5636-3bc1-48b2-ab59-435f3f963176 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:48:35.580991 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a7d5636-3bc1-48b2-ab59-435f3f963176 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:48:35.583023 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 670/1354] 10.222.0.33 () {54 vars in 921 bytes} [Sun Oct 9 00:48:35 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:48:35.588242 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-646b8187-0070-4c60-8808-b9f79a722cbe None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:48:35.588702 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-646b8187-0070-4c60-8808-b9f79a722cbe None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:48:35.588953 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-646b8187-0070-4c60-8808-b9f79a722cbe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:48:35.665442 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-646b8187-0070-4c60-8808-b9f79a722cbe None None] MFA Rules not processed for user `d25e240cb93643ed8122ccac9586936f`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:48:35.666645 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-646b8187-0070-4c60-8808-b9f79a722cbe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:48:35.709411 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:48:35.773324 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 685/1355] 10.222.0.33 () {58 vars in 1018 bytes} [Sun Oct 9 00:48:35 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:48:39.342253 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ad3b3c-2f78-4906-8f75-f6a6861659d7 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:48:39.342791 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ad3b3c-2f78-4906-8f75-f6a6861659d7 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:48:39.342791 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ad3b3c-2f78-4906-8f75-f6a6861659d7 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:48:39.344032 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 671/1356] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:48:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:48:39.349468 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e6f435e-ef43-4725-a283-0c0e54ae8723 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:48:39.349922 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e6f435e-ef43-4725-a283-0c0e54ae8723 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:48:39.350020 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e6f435e-ef43-4725-a283-0c0e54ae8723 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:48:39.427415 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-3e6f435e-ef43-4725-a283-0c0e54ae8723 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:48:39.428927 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-3e6f435e-ef43-4725-a283-0c0e54ae8723 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:48:39.547629 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 686/1357] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:48:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:48:39.842245 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d5a90d9-f8ca-46f5-8ecd-6aba2ba25b88 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:48:39.842916 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d5a90d9-f8ca-46f5-8ecd-6aba2ba25b88 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:48:39.843504 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d5a90d9-f8ca-46f5-8ecd-6aba2ba25b88 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:48:39.846447 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 672/1358] 10.222.0.33 () {54 vars in 961 bytes} [Sun Oct 9 00:48:39 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:48:39.856567 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12c9f315-65f7-4f28-b77e-6f3a6fb8ea51 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:48:39.857478 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12c9f315-65f7-4f28-b77e-6f3a6fb8ea51 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:48:39.858314 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12c9f315-65f7-4f28-b77e-6f3a6fb8ea51 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:48:39.879493 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:48:39.946129 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-12c9f315-65f7-4f28-b77e-6f3a6fb8ea51 None None] MFA Rules not processed for user `e0a5d95969e74157bb2d2a57207e1651`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:48:39.947631 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-12c9f315-65f7-4f28-b77e-6f3a6fb8ea51 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:48:40.088191 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 687/1359] 10.222.0.33 () {58 vars in 1058 bytes} [Sun Oct 9 00:48:39 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 234 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:48:40.096073 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d5a90d9-f8ca-46f5-8ecd-6aba2ba25b88 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:48:40.097763 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-3d5a90d9-f8ca-46f5-8ecd-6aba2ba25b88 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:48:40.261235 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d5a90d9-f8ca-46f5-8ecd-6aba2ba25b88 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:48:40.338181 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:48:40.380343 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60371bbf-81dd-444f-9e24-6fcc405f4cd1 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a5d95969e74157bb2d2a57207e1651', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:48:40.380991 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60371bbf-81dd-444f-9e24-6fcc405f4cd1 service nova] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:48:40.381201 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60371bbf-81dd-444f-9e24-6fcc405f4cd1 service nova] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:48:40.381472 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60371bbf-81dd-444f-9e24-6fcc405f4cd1 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:48:40.383040 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-60371bbf-81dd-444f-9e24-6fcc405f4cd1 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:48:40.481882 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60371bbf-81dd-444f-9e24-6fcc405f4cd1 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:48:40.483463 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60371bbf-81dd-444f-9e24-6fcc405f4cd1 service nova] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:48:40.503425 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 673/1360] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:48:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 408 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:48:42.882907 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c9203fa-ea47-4a21-aa97-0ad778eaeba3 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:48:42.883349 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c9203fa-ea47-4a21-aa97-0ad778eaeba3 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:48:42.883534 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c9203fa-ea47-4a21-aa97-0ad778eaeba3 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:48:42.890182 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 688/1361] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:48:42 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:48:42.891519 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61fc5f34-bd51-48be-a445-7eec7a9e90f9 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:48:42.891519 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61fc5f34-bd51-48be-a445-7eec7a9e90f9 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:48:42.892372 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61fc5f34-bd51-48be-a445-7eec7a9e90f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:48:42.964332 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-61fc5f34-bd51-48be-a445-7eec7a9e90f9 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:48:42.965524 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-61fc5f34-bd51-48be-a445-7eec7a9e90f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:48:43.005709 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:48:43.070161 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 674/1362] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:48:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 09 00:48:43.361657 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c9203fa-ea47-4a21-aa97-0ad778eaeba3 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:48:43.363877 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-8c9203fa-ea47-4a21-aa97-0ad778eaeba3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:48:43.435042 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:48:43.539506 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c9203fa-ea47-4a21-aa97-0ad778eaeba3 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:48:43.665557 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ce8a709-fc79-4efe-b5dd-57b92cf2910e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a5d95969e74157bb2d2a57207e1651', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:48:43.666155 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ce8a709-fc79-4efe-b5dd-57b92cf2910e service nova] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:48:43.666429 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ce8a709-fc79-4efe-b5dd-57b92cf2910e service nova] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:48:43.666705 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ce8a709-fc79-4efe-b5dd-57b92cf2910e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:48:43.668371 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-9ce8a709-fc79-4efe-b5dd-57b92cf2910e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:48:43.692537 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:48:43.766506 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ce8a709-fc79-4efe-b5dd-57b92cf2910e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:48:43.768164 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ce8a709-fc79-4efe-b5dd-57b92cf2910e service nova] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:48:43.787810 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 689/1363] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:48:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 427 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:48:46.278571 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc4cb944-f19a-4a67-8407-347a67578fa6 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:48:46.279090 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc4cb944-f19a-4a67-8407-347a67578fa6 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:48:46.279293 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc4cb944-f19a-4a67-8407-347a67578fa6 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:48:46.280751 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 675/1364] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:48:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:48:46.286384 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa7a7be-2cb1-4900-a588-814f4485858d None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:48:46.286816 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa7a7be-2cb1-4900-a588-814f4485858d None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:48:46.286984 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa7a7be-2cb1-4900-a588-814f4485858d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:48:46.355884 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-9fa7a7be-2cb1-4900-a588-814f4485858d None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:48:46.357082 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-9fa7a7be-2cb1-4900-a588-814f4485858d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:48:46.480091 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:48:46.484053 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 690/1365] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:48:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 199 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:48:46.771267 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13318bb8-1a29-42a7-9b04-2f95baffc2b8 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:48:46.771652 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13318bb8-1a29-42a7-9b04-2f95baffc2b8 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:48:46.772003 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13318bb8-1a29-42a7-9b04-2f95baffc2b8 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:48:46.773472 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 676/1366] 10.222.0.33 () {54 vars in 961 bytes} [Sun Oct 9 00:48:46 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:48:46.778699 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5409720c-569c-4a1e-9215-aeb699bfa81f None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:48:46.778981 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5409720c-569c-4a1e-9215-aeb699bfa81f None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:48:46.779255 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5409720c-569c-4a1e-9215-aeb699bfa81f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:48:46.858481 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-5409720c-569c-4a1e-9215-aeb699bfa81f None None] MFA Rules not processed for user `e0a5d95969e74157bb2d2a57207e1651`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:48:46.859673 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-5409720c-569c-4a1e-9215-aeb699bfa81f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:48:46.981014 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 691/1367] 10.222.0.33 () {58 vars in 1058 bytes} [Sun Oct 9 00:48:46 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 203 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:48:46.989413 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13318bb8-1a29-42a7-9b04-2f95baffc2b8 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:48:46.991039 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-13318bb8-1a29-42a7-9b04-2f95baffc2b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:48:47.158140 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13318bb8-1a29-42a7-9b04-2f95baffc2b8 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:48:47.180921 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:48:47.287906 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e4874c6-6b94-461b-a51a-058e72520c24 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a5d95969e74157bb2d2a57207e1651', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:48:47.288589 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e4874c6-6b94-461b-a51a-058e72520c24 service nova] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:48:47.289030 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e4874c6-6b94-461b-a51a-058e72520c24 service nova] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:48:47.289312 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e4874c6-6b94-461b-a51a-058e72520c24 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:48:47.290911 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-4e4874c6-6b94-461b-a51a-058e72520c24 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:48:47.390359 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e4874c6-6b94-461b-a51a-058e72520c24 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:48:47.392181 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e4874c6-6b94-461b-a51a-058e72520c24 service nova] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:48:47.414469 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 677/1368] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:48:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 425 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:48:47.505076 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-206184e5-91ec-4e4b-84da-339e984bf0da None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:48:47.505630 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-206184e5-91ec-4e4b-84da-339e984bf0da None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:48:47.506388 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-206184e5-91ec-4e4b-84da-339e984bf0da None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:48:47.508065 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 692/1369] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:48:47 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:48:47.514174 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea30bbd-e8f1-4bbc-ad37-c8adfd6a29e8 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:48:47.514564 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea30bbd-e8f1-4bbc-ad37-c8adfd6a29e8 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:48:47.514907 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea30bbd-e8f1-4bbc-ad37-c8adfd6a29e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:48:47.558255 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:48:47.603179 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-fea30bbd-e8f1-4bbc-ad37-c8adfd6a29e8 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:48:47.604604 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-fea30bbd-e8f1-4bbc-ad37-c8adfd6a29e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:48:47.719621 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 678/1370] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:48:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 206 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:48:48.063003 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d12acacb-96f3-47d7-8da9-daaab63099bf None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:48:48.063367 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d12acacb-96f3-47d7-8da9-daaab63099bf None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:48:48.063748 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d12acacb-96f3-47d7-8da9-daaab63099bf None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:48:48.065323 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 693/1371] 10.222.0.33 () {54 vars in 963 bytes} [Sun Oct 9 00:48:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:48:48.070672 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfc5e9e4-a20f-4861-8360-67e32aa4cdb7 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:48:48.071031 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfc5e9e4-a20f-4861-8360-67e32aa4cdb7 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:48:48.071374 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfc5e9e4-a20f-4861-8360-67e32aa4cdb7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:48:48.163996 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-bfc5e9e4-a20f-4861-8360-67e32aa4cdb7 None None] MFA Rules not processed for user `f3f48335668a4e8f934ac11ff66cd308`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:48:48.165823 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-bfc5e9e4-a20f-4861-8360-67e32aa4cdb7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:48:48.214661 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:48:48.246395 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 679/1372] 10.222.0.33 () {58 vars in 1060 bytes} [Sun Oct 9 00:48:48 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:48:48.254531 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d12acacb-96f3-47d7-8da9-daaab63099bf None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:48:48.256059 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-d12acacb-96f3-47d7-8da9-daaab63099bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:48:48.309367 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:48:48.370785 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d12acacb-96f3-47d7-8da9-daaab63099bf None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:48:48.459707 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90cc6074-7a27-45bc-8b25-1827113ceb2b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3f48335668a4e8f934ac11ff66cd308', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:48:48.460260 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90cc6074-7a27-45bc-8b25-1827113ceb2b service cinder] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:48:48.460538 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90cc6074-7a27-45bc-8b25-1827113ceb2b service cinder] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:48:48.460803 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90cc6074-7a27-45bc-8b25-1827113ceb2b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:48:48.462355 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-90cc6074-7a27-45bc-8b25-1827113ceb2b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:48:48.648560 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:48:48.676286 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90cc6074-7a27-45bc-8b25-1827113ceb2b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:48:48.681084 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90cc6074-7a27-45bc-8b25-1827113ceb2b service cinder] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:48:48.731580 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 694/1373] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:48:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 478 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:48:49.698051 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58842553-f1b8-4cf3-9bd9-a9ad9f313339 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:48:49.698584 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58842553-f1b8-4cf3-9bd9-a9ad9f313339 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:48:49.698584 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58842553-f1b8-4cf3-9bd9-a9ad9f313339 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:48:49.700140 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 680/1374] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:48:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:48:49.705061 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2de5730-54d1-4b53-a12f-2db109089ee6 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:48:49.705447 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2de5730-54d1-4b53-a12f-2db109089ee6 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:48:49.705742 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2de5730-54d1-4b53-a12f-2db109089ee6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:48:49.771502 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-a2de5730-54d1-4b53-a12f-2db109089ee6 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:48:49.772731 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-a2de5730-54d1-4b53-a12f-2db109089ee6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:48:49.870408 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 695/1375] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:48:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:48:50.162230 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58842553-f1b8-4cf3-9bd9-a9ad9f313339 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:48:50.162230 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-58842553-f1b8-4cf3-9bd9-a9ad9f313339 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:48:50.328344 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58842553-f1b8-4cf3-9bd9-a9ad9f313339 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:48:50.385041 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:48:50.455762 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5130316d-afc1-4fe8-a0bc-068decff5255 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a5d95969e74157bb2d2a57207e1651', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:48:50.456387 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5130316d-afc1-4fe8-a0bc-068decff5255 service nova] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:48:50.456666 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5130316d-afc1-4fe8-a0bc-068decff5255 service nova] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:48:50.456971 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5130316d-afc1-4fe8-a0bc-068decff5255 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:48:50.459375 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-5130316d-afc1-4fe8-a0bc-068decff5255 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:48:50.554504 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5130316d-afc1-4fe8-a0bc-068decff5255 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:48:50.556135 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5130316d-afc1-4fe8-a0bc-068decff5255 service nova] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:48:50.575397 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 681/1376] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:48:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 417 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:48:52.800560 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f8d1a55-97ae-4956-a5e2-865ad5c8731f None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:48:52.801066 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f8d1a55-97ae-4956-a5e2-865ad5c8731f None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:48:52.801066 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f8d1a55-97ae-4956-a5e2-865ad5c8731f None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:48:52.802685 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 696/1377] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:48:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:48:52.807883 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86ed9731-a47d-4ff9-a582-3031363a6052 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:48:52.808337 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86ed9731-a47d-4ff9-a582-3031363a6052 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:48:52.808443 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86ed9731-a47d-4ff9-a582-3031363a6052 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:48:52.881868 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-86ed9731-a47d-4ff9-a582-3031363a6052 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:48:52.883086 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-86ed9731-a47d-4ff9-a582-3031363a6052 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:48:52.891018 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:48:52.992244 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 682/1378] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:48:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:48:53.283005 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f8d1a55-97ae-4956-a5e2-865ad5c8731f None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:48:53.285128 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-5f8d1a55-97ae-4956-a5e2-865ad5c8731f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:48:53.351671 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:48:53.441434 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f8d1a55-97ae-4956-a5e2-865ad5c8731f None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:48:53.560202 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c25f213-9f26-4934-990f-79a15bb1b1f9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a5d95969e74157bb2d2a57207e1651', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:48:53.560802 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c25f213-9f26-4934-990f-79a15bb1b1f9 service nova] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:48:53.561079 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c25f213-9f26-4934-990f-79a15bb1b1f9 service nova] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:48:53.561348 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c25f213-9f26-4934-990f-79a15bb1b1f9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:48:53.562943 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-5c25f213-9f26-4934-990f-79a15bb1b1f9 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:48:53.582168 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:48:53.658519 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c25f213-9f26-4934-990f-79a15bb1b1f9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:48:53.660064 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c25f213-9f26-4934-990f-79a15bb1b1f9 service nova] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:48:53.679335 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 697/1379] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:48:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 397 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:48:55.884789 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-446ef23b-5a94-4947-beba-d6d34aac51de None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:48:55.885382 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-446ef23b-5a94-4947-beba-d6d34aac51de None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:48:55.885382 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-446ef23b-5a94-4947-beba-d6d34aac51de None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:48:55.886768 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 683/1380] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:48:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:48:55.891997 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-582ae542-d24f-489b-a5ab-ce72cac7f3d2 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:48:55.892425 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-582ae542-d24f-489b-a5ab-ce72cac7f3d2 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:48:55.892548 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-582ae542-d24f-489b-a5ab-ce72cac7f3d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:48:55.962484 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-582ae542-d24f-489b-a5ab-ce72cac7f3d2 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:48:55.963719 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-582ae542-d24f-489b-a5ab-ce72cac7f3d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:48:56.077000 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:48:56.089245 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 698/1381] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:48:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 198 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:48:56.382000 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-446ef23b-5a94-4947-beba-d6d34aac51de None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:48:56.383507 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-446ef23b-5a94-4947-beba-d6d34aac51de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:48:56.520003 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:48:56.531375 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-446ef23b-5a94-4947-beba-d6d34aac51de None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:48:56.663877 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e04c9044-c685-4b59-8799-42fef368b615 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a5d95969e74157bb2d2a57207e1651', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:48:56.664488 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e04c9044-c685-4b59-8799-42fef368b615 service nova] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:48:56.664764 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e04c9044-c685-4b59-8799-42fef368b615 service nova] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:48:56.665042 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e04c9044-c685-4b59-8799-42fef368b615 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:48:56.666643 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-e04c9044-c685-4b59-8799-42fef368b615 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:48:56.757745 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e04c9044-c685-4b59-8799-42fef368b615 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:48:56.759246 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e04c9044-c685-4b59-8799-42fef368b615 service nova] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:48:56.772054 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:48:56.784464 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 684/1382] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:48:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 403 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 09 00:48:59.720307 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb7f494-a249-4288-b8b9-5391a11bd0b8 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:48:59.720847 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb7f494-a249-4288-b8b9-5391a11bd0b8 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:48:59.720847 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb7f494-a249-4288-b8b9-5391a11bd0b8 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:48:59.722213 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 699/1383] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:48:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:48:59.729067 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e133ffe-a66c-4866-81cf-dfbb2fe45580 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:48:59.729654 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e133ffe-a66c-4866-81cf-dfbb2fe45580 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:48:59.729654 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e133ffe-a66c-4866-81cf-dfbb2fe45580 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:48:59.800167 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-2e133ffe-a66c-4866-81cf-dfbb2fe45580 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:48:59.801432 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-2e133ffe-a66c-4866-81cf-dfbb2fe45580 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:48:59.913317 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 685/1384] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:48:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:49:00.253270 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcb7f494-a249-4288-b8b9-5391a11bd0b8 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:49:00.255313 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-dcb7f494-a249-4288-b8b9-5391a11bd0b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:49:00.409410 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcb7f494-a249-4288-b8b9-5391a11bd0b8 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:49:00.478073 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:49:00.529826 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-931d236b-05f9-4595-8505-61df68de2411 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a5d95969e74157bb2d2a57207e1651', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:49:00.530428 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-931d236b-05f9-4595-8505-61df68de2411 service nova] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:49:00.530703 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-931d236b-05f9-4595-8505-61df68de2411 service nova] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:49:00.531016 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-931d236b-05f9-4595-8505-61df68de2411 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:49:00.532879 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-931d236b-05f9-4595-8505-61df68de2411 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:49:00.635686 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-931d236b-05f9-4595-8505-61df68de2411 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:49:00.637527 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-931d236b-05f9-4595-8505-61df68de2411 service nova] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:49:00.658353 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 700/1385] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:49:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 406 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:49:02.804106 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8f670b9-43d9-45c0-b93b-be5021ea0912 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:49:02.804639 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8f670b9-43d9-45c0-b93b-be5021ea0912 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:49:02.804639 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8f670b9-43d9-45c0-b93b-be5021ea0912 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:49:02.806121 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 686/1386] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:49:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:49:02.811967 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eacdf3e0-dd30-496a-b741-9c099583f5f4 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:49:02.812376 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eacdf3e0-dd30-496a-b741-9c099583f5f4 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:49:02.812656 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eacdf3e0-dd30-496a-b741-9c099583f5f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:49:02.883138 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-eacdf3e0-dd30-496a-b741-9c099583f5f4 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:49:02.884469 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-eacdf3e0-dd30-496a-b741-9c099583f5f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:49:02.917627 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:49:02.984982 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 701/1387] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:49:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:49:03.275562 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8f670b9-43d9-45c0-b93b-be5021ea0912 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:49:03.277091 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-c8f670b9-43d9-45c0-b93b-be5021ea0912 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:49:03.353072 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:49:03.430238 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8f670b9-43d9-45c0-b93b-be5021ea0912 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:49:03.543941 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72ef5217-9f71-44e1-a4f6-817e2968315e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a5d95969e74157bb2d2a57207e1651', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:49:03.544622 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ef5217-9f71-44e1-a4f6-817e2968315e service nova] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:49:03.544809 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ef5217-9f71-44e1-a4f6-817e2968315e service nova] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:49:03.545086 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ef5217-9f71-44e1-a4f6-817e2968315e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:49:03.546689 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-72ef5217-9f71-44e1-a4f6-817e2968315e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:49:03.589468 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:49:03.647147 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72ef5217-9f71-44e1-a4f6-817e2968315e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:49:03.648761 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72ef5217-9f71-44e1-a4f6-817e2968315e service nova] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:49:03.668518 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 687/1388] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:49:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 393 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:49:05.865882 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-758f0995-fea7-43cf-a997-12794c31d47e None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:49:05.866412 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-758f0995-fea7-43cf-a997-12794c31d47e None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:49:05.866412 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-758f0995-fea7-43cf-a997-12794c31d47e None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:49:05.867784 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 702/1389] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:49:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:49:05.873172 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d5a0b2-71e9-485d-b2a1-78c6f42eba22 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:49:05.873606 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d5a0b2-71e9-485d-b2a1-78c6f42eba22 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:49:05.873710 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d5a0b2-71e9-485d-b2a1-78c6f42eba22 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:49:05.941309 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-d7d5a0b2-71e9-485d-b2a1-78c6f42eba22 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:49:05.942606 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-d7d5a0b2-71e9-485d-b2a1-78c6f42eba22 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:49:06.040367 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 688/1390] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:49:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 09 00:49:06.329831 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-758f0995-fea7-43cf-a997-12794c31d47e None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:49:06.331404 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-758f0995-fea7-43cf-a997-12794c31d47e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:49:06.481672 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-758f0995-fea7-43cf-a997-12794c31d47e None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:49:06.498411 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:49:06.597338 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-190a4c29-b2be-44df-9314-264c87c3e892 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a5d95969e74157bb2d2a57207e1651', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:49:06.597866 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-190a4c29-b2be-44df-9314-264c87c3e892 service nova] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:49:06.598129 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-190a4c29-b2be-44df-9314-264c87c3e892 service nova] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:49:06.598400 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-190a4c29-b2be-44df-9314-264c87c3e892 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:49:06.600590 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-190a4c29-b2be-44df-9314-264c87c3e892 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:49:06.688592 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-190a4c29-b2be-44df-9314-264c87c3e892 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:49:06.690188 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-190a4c29-b2be-44df-9314-264c87c3e892 service nova] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:49:06.707824 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 703/1391] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:49:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 378 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:49:07.398987 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e53ad0fb-5385-4d2a-9083-c3bdc4b4c16a None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:49:07.399519 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e53ad0fb-5385-4d2a-9083-c3bdc4b4c16a None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:49:07.399519 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e53ad0fb-5385-4d2a-9083-c3bdc4b4c16a None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:49:07.400815 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 689/1392] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:49:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:49:07.406315 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea27b5f-ee49-4018-a743-2731faa10a00 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:49:07.406737 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea27b5f-ee49-4018-a743-2731faa10a00 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:49:07.406857 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea27b5f-ee49-4018-a743-2731faa10a00 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:49:07.432421 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:49:07.475697 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-3ea27b5f-ee49-4018-a743-2731faa10a00 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:49:07.476891 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-3ea27b5f-ee49-4018-a743-2731faa10a00 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:49:07.580913 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 704/1393] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:49:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:49:07.885519 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd3e51f4-b0d9-4c4d-aa6c-168df999f8d3 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:49:07.885918 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd3e51f4-b0d9-4c4d-aa6c-168df999f8d3 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:49:07.886294 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd3e51f4-b0d9-4c4d-aa6c-168df999f8d3 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:49:07.887870 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 690/1394] 10.222.0.33 () {54 vars in 962 bytes} [Sun Oct 9 00:49:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:49:07.893941 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-054d6849-476a-41b4-98f4-f4b2e592385a None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:49:07.894303 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-054d6849-476a-41b4-98f4-f4b2e592385a None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:49:07.894675 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-054d6849-476a-41b4-98f4-f4b2e592385a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:49:07.977016 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-054d6849-476a-41b4-98f4-f4b2e592385a None None] MFA Rules not processed for user `1f7a88094d73471787cc0a287adbd394`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:49:07.978235 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-054d6849-476a-41b4-98f4-f4b2e592385a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:49:08.023415 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:49:08.063539 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 705/1395] 10.222.0.33 () {58 vars in 1059 bytes} [Sun Oct 9 00:49:07 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:49:08.075639 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd3e51f4-b0d9-4c4d-aa6c-168df999f8d3 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:49:08.075639 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-bd3e51f4-b0d9-4c4d-aa6c-168df999f8d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:49:08.088927 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:49:08.177259 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd3e51f4-b0d9-4c4d-aa6c-168df999f8d3 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:49:08.249299 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60e8fbf0-47bb-4369-80a2-e4dc86b6a88c service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1f7a88094d73471787cc0a287adbd394', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:49:08.249894 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e8fbf0-47bb-4369-80a2-e4dc86b6a88c service glance] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:49:08.250176 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e8fbf0-47bb-4369-80a2-e4dc86b6a88c service glance] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:49:08.250445 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e8fbf0-47bb-4369-80a2-e4dc86b6a88c service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:49:08.252042 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-60e8fbf0-47bb-4369-80a2-e4dc86b6a88c service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:49:08.319470 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:49:08.361252 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60e8fbf0-47bb-4369-80a2-e4dc86b6a88c service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:49:08.362688 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60e8fbf0-47bb-4369-80a2-e4dc86b6a88c service glance] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:49:08.381157 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 691/1396] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:49:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:49:08.907081 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d9e5fb-9baf-465b-b15a-b59c22df87d5 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:49:08.907576 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d9e5fb-9baf-465b-b15a-b59c22df87d5 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:49:08.907718 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d9e5fb-9baf-465b-b15a-b59c22df87d5 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:49:08.909000 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 706/1397] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:49:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:49:08.914785 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b63333-9770-4cd1-a3f9-7718a7555703 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:49:08.915176 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b63333-9770-4cd1-a3f9-7718a7555703 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:49:08.915347 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b63333-9770-4cd1-a3f9-7718a7555703 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:49:08.990835 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-05b63333-9770-4cd1-a3f9-7718a7555703 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:49:08.992117 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-05b63333-9770-4cd1-a3f9-7718a7555703 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:49:09.094777 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 692/1398] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:49:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:49:09.124658 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056895b6-4e16-444f-a393-e945516f171c None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:49:09.125008 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056895b6-4e16-444f-a393-e945516f171c None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:49:09.125351 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056895b6-4e16-444f-a393-e945516f171c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:49:09.191980 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-056895b6-4e16-444f-a393-e945516f171c None None] MFA Rules not processed for user `5f3247ddcda54394995d844c776bf43c`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:49:09.193693 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-056895b6-4e16-444f-a393-e945516f171c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:49:09.282512 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 707/1399] 10.222.0.33 () {58 vars in 1009 bytes} [Sun Oct 9 00:49:09 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:49:09.328486 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d19642fc-52a2-4300-9118-b8118e679ae0 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:49:09.328777 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d19642fc-52a2-4300-9118-b8118e679ae0 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:49:09.329054 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d19642fc-52a2-4300-9118-b8118e679ae0 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:49:09.330420 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 693/1400] 10.222.0.33 () {54 vars in 943 bytes} [Sun Oct 9 00:49:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:49:09.337604 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a81beef-f21f-417d-9c72-ed43b3ad2278 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:49:09.337968 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a81beef-f21f-417d-9c72-ed43b3ad2278 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:49:09.338318 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a81beef-f21f-417d-9c72-ed43b3ad2278 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:49:09.385431 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d19642fc-52a2-4300-9118-b8118e679ae0 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:49:09.386949 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-d19642fc-52a2-4300-9118-b8118e679ae0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:49:09.409229 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:49:09.412215 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:49:09.417190 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-4a81beef-f21f-417d-9c72-ed43b3ad2278 None None] MFA Rules not processed for user `70e367e802894b08afd7d756d8d9ecff`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:49:09.418406 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-4a81beef-f21f-417d-9c72-ed43b3ad2278 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:49:09.526164 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 708/1401] 10.222.0.33 () {58 vars in 1040 bytes} [Sun Oct 9 00:49:09 2022] POST /identity/v3/auth/tokens => generated 3304 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:49:09.535672 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a81beef-f21f-417d-9c72-ed43b3ad2278 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:49:09.536481 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d19642fc-52a2-4300-9118-b8118e679ae0 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:49:09.545226 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a81beef-f21f-417d-9c72-ed43b3ad2278 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:49:09.632643 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:49:09.651852 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-866be638-00e5-4501-a83c-a1a36e764f91 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a5d95969e74157bb2d2a57207e1651', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:49:09.652510 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-866be638-00e5-4501-a83c-a1a36e764f91 service nova] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:49:09.652764 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-866be638-00e5-4501-a83c-a1a36e764f91 service nova] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:49:09.652861 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-944e25c3-87d4-4a75-9ec3-e0bb7353bc0b service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '70e367e802894b08afd7d756d8d9ecff', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:49:09.653036 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-866be638-00e5-4501-a83c-a1a36e764f91 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:49:09.653366 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-944e25c3-87d4-4a75-9ec3-e0bb7353bc0b service swift] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:49:09.653648 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-944e25c3-87d4-4a75-9ec3-e0bb7353bc0b service swift] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:49:09.653924 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-944e25c3-87d4-4a75-9ec3-e0bb7353bc0b service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:49:09.654644 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-866be638-00e5-4501-a83c-a1a36e764f91 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:49:09.655772 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-944e25c3-87d4-4a75-9ec3-e0bb7353bc0b service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:49:09.658292 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:49:09.746476 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-944e25c3-87d4-4a75-9ec3-e0bb7353bc0b service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:49:09.749247 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-944e25c3-87d4-4a75-9ec3-e0bb7353bc0b service swift] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:49:09.750051 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-866be638-00e5-4501-a83c-a1a36e764f91 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:49:09.751689 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-866be638-00e5-4501-a83c-a1a36e764f91 service nova] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:49:09.758038 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 709/1402] 10.222.0.33 () {60 vars in 1384 bytes} [Sun Oct 9 00:49:09 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 223 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Oct 09 00:49:09.772560 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 694/1403] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:49:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 388 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 09 00:49:12.019637 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7648bc81-ceca-4a36-9b70-7b2e0565b10e None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:49:12.019637 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7648bc81-ceca-4a36-9b70-7b2e0565b10e None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:49:12.019637 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7648bc81-ceca-4a36-9b70-7b2e0565b10e None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:49:12.025026 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 710/1404] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:49:12 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:49:12.032817 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b0a6deb-3f99-4fb4-a33a-62e00bf759d9 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:49:12.032817 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b0a6deb-3f99-4fb4-a33a-62e00bf759d9 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:49:12.032817 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b0a6deb-3f99-4fb4-a33a-62e00bf759d9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:49:12.114155 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-4b0a6deb-3f99-4fb4-a33a-62e00bf759d9 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:49:12.117511 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-4b0a6deb-3f99-4fb4-a33a-62e00bf759d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:49:12.182046 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:49:12.231037 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 695/1405] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:49:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 206 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:49:12.528454 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7648bc81-ceca-4a36-9b70-7b2e0565b10e None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:49:12.530003 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-7648bc81-ceca-4a36-9b70-7b2e0565b10e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:49:12.688542 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:49:12.691319 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7648bc81-ceca-4a36-9b70-7b2e0565b10e None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:49:12.818273 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4473a0e5-1d15-41d7-8f63-a5b0cc04b1da service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a5d95969e74157bb2d2a57207e1651', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:49:12.819110 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4473a0e5-1d15-41d7-8f63-a5b0cc04b1da service nova] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:49:12.819393 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4473a0e5-1d15-41d7-8f63-a5b0cc04b1da service nova] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:49:12.819690 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4473a0e5-1d15-41d7-8f63-a5b0cc04b1da service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:49:12.821326 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-4473a0e5-1d15-41d7-8f63-a5b0cc04b1da service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:49:12.931631 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4473a0e5-1d15-41d7-8f63-a5b0cc04b1da service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:49:12.933641 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4473a0e5-1d15-41d7-8f63-a5b0cc04b1da service nova] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:49:12.953849 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 711/1406] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:49:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 426 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:49:15.145595 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e85b01c-4f68-49e5-8509-5e7d9e178a0a None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:49:15.147119 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e85b01c-4f68-49e5-8509-5e7d9e178a0a None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:49:15.147659 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e85b01c-4f68-49e5-8509-5e7d9e178a0a None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:49:15.149610 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 696/1407] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:49:15 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:49:15.154974 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b82bd47-74c9-45cd-89c8-e4b582e63af7 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:49:15.155460 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b82bd47-74c9-45cd-89c8-e4b582e63af7 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:49:15.155557 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b82bd47-74c9-45cd-89c8-e4b582e63af7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:49:15.160257 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:49:15.227895 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-9b82bd47-74c9-45cd-89c8-e4b582e63af7 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:49:15.229458 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-9b82bd47-74c9-45cd-89c8-e4b582e63af7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:49:15.341829 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 712/1408] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:49:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:49:15.634981 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e85b01c-4f68-49e5-8509-5e7d9e178a0a None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:49:15.636830 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-9e85b01c-4f68-49e5-8509-5e7d9e178a0a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:49:15.800371 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e85b01c-4f68-49e5-8509-5e7d9e178a0a None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:49:15.844636 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:49:15.934623 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1946ec00-93d1-447e-9631-4ba2c273ac57 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a5d95969e74157bb2d2a57207e1651', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:49:15.934623 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1946ec00-93d1-447e-9631-4ba2c273ac57 service nova] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:49:15.934623 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1946ec00-93d1-447e-9631-4ba2c273ac57 service nova] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:49:15.934623 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1946ec00-93d1-447e-9631-4ba2c273ac57 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:49:15.951076 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-1946ec00-93d1-447e-9631-4ba2c273ac57 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:49:16.085577 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1946ec00-93d1-447e-9631-4ba2c273ac57 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:49:16.087219 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1946ec00-93d1-447e-9631-4ba2c273ac57 service nova] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:49:16.116093 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 697/1409] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:49:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 481 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:49:25.340260 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=64 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:49:26.111383 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=79 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:49:32.332528 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a3b9d70-6a78-4150-9d3f-09b42520578b None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:49:32.333060 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a3b9d70-6a78-4150-9d3f-09b42520578b None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:49:32.333060 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a3b9d70-6a78-4150-9d3f-09b42520578b None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:49:32.334437 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 713/1410] 10.222.0.33 () {54 vars in 918 bytes} [Sun Oct 9 00:49:32 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:49:32.339387 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae6d18b-ee35-4e2e-8fff-3b64729fa4af None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:49:32.339872 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae6d18b-ee35-4e2e-8fff-3b64729fa4af None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:49:32.339976 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae6d18b-ee35-4e2e-8fff-3b64729fa4af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:49:32.414971 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-2ae6d18b-ee35-4e2e-8fff-3b64729fa4af None None] MFA Rules not processed for user `d25e240cb93643ed8122ccac9586936f`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:49:32.416184 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-2ae6d18b-ee35-4e2e-8fff-3b64729fa4af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:49:32.520201 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 698/1411] 10.222.0.33 () {58 vars in 1015 bytes} [Sun Oct 9 00:49:32 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:49:32.534002 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4445b366-a497-4836-a895-2c22fc57b3d0 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:49:32.534285 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4445b366-a497-4836-a895-2c22fc57b3d0 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:49:32.534557 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4445b366-a497-4836-a895-2c22fc57b3d0 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:49:32.535904 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 714/1412] 10.222.0.33 () {54 vars in 918 bytes} [Sun Oct 9 00:49:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:49:32.540297 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e1dd30-ace7-4ef4-b0d0-c5326cc2e87c None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:49:32.540588 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e1dd30-ace7-4ef4-b0d0-c5326cc2e87c None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:49:32.540867 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e1dd30-ace7-4ef4-b0d0-c5326cc2e87c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:49:32.612829 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:49:32.616148 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-83e1dd30-ace7-4ef4-b0d0-c5326cc2e87c None None] MFA Rules not processed for user `e0a5d95969e74157bb2d2a57207e1651`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:49:32.617988 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-83e1dd30-ace7-4ef4-b0d0-c5326cc2e87c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:49:32.730441 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 699/1413] 10.222.0.33 () {58 vars in 1015 bytes} [Sun Oct 9 00:49:32 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:49:35.276612 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-460c1cc8-1d6c-457d-bfe3-b759c186c54e None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:49:35.277173 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-460c1cc8-1d6c-457d-bfe3-b759c186c54e None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:49:35.277173 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-460c1cc8-1d6c-457d-bfe3-b759c186c54e None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:49:35.278490 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 715/1414] 127.0.0.1 () {54 vars in 917 bytes} [Sun Oct 9 00:49:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:49:35.283656 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd5d5519-0e80-43b0-ab7c-ee42b8f21314 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:49:35.284102 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd5d5519-0e80-43b0-ab7c-ee42b8f21314 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:49:35.284216 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd5d5519-0e80-43b0-ab7c-ee42b8f21314 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:49:35.349893 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-cd5d5519-0e80-43b0-ab7c-ee42b8f21314 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:49:35.351011 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-cd5d5519-0e80-43b0-ab7c-ee42b8f21314 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:49:35.404618 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:49:35.446038 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 700/1415] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Oct 9 00:49:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 09 00:49:35.735765 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-460c1cc8-1d6c-457d-bfe3-b759c186c54e None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:49:35.738141 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-460c1cc8-1d6c-457d-bfe3-b759c186c54e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:49:35.883636 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-460c1cc8-1d6c-457d-bfe3-b759c186c54e None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:49:35.956832 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:49:35.996268 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d92ee8d-5e96-45e0-80fd-dcb3ff29e3a3 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a5d95969e74157bb2d2a57207e1651', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:49:35.996801 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d92ee8d-5e96-45e0-80fd-dcb3ff29e3a3 service nova] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:49:35.997080 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d92ee8d-5e96-45e0-80fd-dcb3ff29e3a3 service nova] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:49:35.997353 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d92ee8d-5e96-45e0-80fd-dcb3ff29e3a3 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:49:35.998934 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-7d92ee8d-5e96-45e0-80fd-dcb3ff29e3a3 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:49:36.088197 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d92ee8d-5e96-45e0-80fd-dcb3ff29e3a3 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:49:36.089832 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d92ee8d-5e96-45e0-80fd-dcb3ff29e3a3 service nova] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:49:36.107986 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 716/1416] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:49:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 373 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:49:37.240340 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0049331-6c4d-4aa9-8fde-e7abe4e396b4 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:49:37.240901 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0049331-6c4d-4aa9-8fde-e7abe4e396b4 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:49:37.240901 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0049331-6c4d-4aa9-8fde-e7abe4e396b4 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:49:37.242231 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 701/1417] 127.0.0.1 () {54 vars in 901 bytes} [Sun Oct 9 00:49:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:49:37.248340 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c16b53c-b2de-4de3-930d-4323f5844d7f None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:49:37.248779 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c16b53c-b2de-4de3-930d-4323f5844d7f None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:49:37.248883 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c16b53c-b2de-4de3-930d-4323f5844d7f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:49:37.324694 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-5c16b53c-b2de-4de3-930d-4323f5844d7f None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:49:37.325893 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-5c16b53c-b2de-4de3-930d-4323f5844d7f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:49:37.352978 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:49:37.421619 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 717/1418] 127.0.0.1 () {58 vars in 998 bytes} [Sun Oct 9 00:49:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:49:37.576294 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0049331-6c4d-4aa9-8fde-e7abe4e396b4 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:49:37.577856 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-e0049331-6c4d-4aa9-8fde-e7abe4e396b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:49:37.720590 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0049331-6c4d-4aa9-8fde-e7abe4e396b4 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:49:37.763227 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:49:37.838427 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-171fc1ac-85bc-4b3d-b4a0-56b45aea6b8c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a5d95969e74157bb2d2a57207e1651', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:49:37.839029 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-171fc1ac-85bc-4b3d-b4a0-56b45aea6b8c service nova] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:49:37.839309 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-171fc1ac-85bc-4b3d-b4a0-56b45aea6b8c service nova] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:49:37.839730 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-171fc1ac-85bc-4b3d-b4a0-56b45aea6b8c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:49:37.841208 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-171fc1ac-85bc-4b3d-b4a0-56b45aea6b8c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:49:37.930718 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-171fc1ac-85bc-4b3d-b4a0-56b45aea6b8c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:49:37.932620 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-171fc1ac-85bc-4b3d-b4a0-56b45aea6b8c service nova] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:49:37.950298 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 702/1419] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:49:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 375 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 09 00:49:39.082754 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cefdc22-06c0-4b2b-8fe2-74d14553e252 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:49:39.083236 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cefdc22-06c0-4b2b-8fe2-74d14553e252 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:49:39.083333 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cefdc22-06c0-4b2b-8fe2-74d14553e252 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:49:39.084833 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 718/1420] 127.0.0.1 () {54 vars in 901 bytes} [Sun Oct 9 00:49:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:49:39.091480 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2efa19e-7e02-4d80-ba3f-5fdfee4f697f None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:49:39.091964 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2efa19e-7e02-4d80-ba3f-5fdfee4f697f None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:49:39.092143 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2efa19e-7e02-4d80-ba3f-5fdfee4f697f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:49:39.145924 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:49:39.168099 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-f2efa19e-7e02-4d80-ba3f-5fdfee4f697f None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:49:39.169318 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-f2efa19e-7e02-4d80-ba3f-5fdfee4f697f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:49:39.269645 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 703/1421] 127.0.0.1 () {58 vars in 998 bytes} [Sun Oct 9 00:49:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:49:39.421187 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cefdc22-06c0-4b2b-8fe2-74d14553e252 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:49:39.422754 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-8cefdc22-06c0-4b2b-8fe2-74d14553e252 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:49:39.566828 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cefdc22-06c0-4b2b-8fe2-74d14553e252 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:49:39.579277 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:49:39.678466 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef438608-1a21-408b-9ac9-b7180f673943 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a5d95969e74157bb2d2a57207e1651', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:49:39.679056 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef438608-1a21-408b-9ac9-b7180f673943 service nova] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:49:39.679331 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef438608-1a21-408b-9ac9-b7180f673943 service nova] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:49:39.679623 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef438608-1a21-408b-9ac9-b7180f673943 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:49:39.681654 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-ef438608-1a21-408b-9ac9-b7180f673943 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:49:39.772254 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef438608-1a21-408b-9ac9-b7180f673943 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:49:39.773872 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef438608-1a21-408b-9ac9-b7180f673943 service nova] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:49:39.792104 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 719/1422] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:49:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 371 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:49:40.899110 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a6186e-44a6-426a-b1eb-461cbc460566 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:49:40.899732 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a6186e-44a6-426a-b1eb-461cbc460566 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:49:40.899732 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a6186e-44a6-426a-b1eb-461cbc460566 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:49:40.900945 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 704/1423] 127.0.0.1 () {54 vars in 901 bytes} [Sun Oct 9 00:49:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:49:40.907934 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea5412a0-9d9e-4490-96fa-bd9af824a080 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:49:40.908397 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea5412a0-9d9e-4490-96fa-bd9af824a080 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:49:40.908505 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea5412a0-9d9e-4490-96fa-bd9af824a080 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:49:40.927758 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:49:40.984786 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-ea5412a0-9d9e-4490-96fa-bd9af824a080 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:49:40.985980 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-ea5412a0-9d9e-4490-96fa-bd9af824a080 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:49:41.081304 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 720/1424] 127.0.0.1 () {58 vars in 998 bytes} [Sun Oct 9 00:49:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:49:41.231364 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62a6186e-44a6-426a-b1eb-461cbc460566 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:49:41.232932 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-62a6186e-44a6-426a-b1eb-461cbc460566 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:49:41.343357 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:49:41.378668 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62a6186e-44a6-426a-b1eb-461cbc460566 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:49:41.492821 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27d28b89-ed2b-4235-836a-fd70777d8bbd service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a5d95969e74157bb2d2a57207e1651', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:49:41.493346 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27d28b89-ed2b-4235-836a-fd70777d8bbd service nova] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:49:41.493625 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27d28b89-ed2b-4235-836a-fd70777d8bbd service nova] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:49:41.493894 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27d28b89-ed2b-4235-836a-fd70777d8bbd service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:49:41.495482 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-27d28b89-ed2b-4235-836a-fd70777d8bbd service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:49:41.571073 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:49:41.585546 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27d28b89-ed2b-4235-836a-fd70777d8bbd service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:49:41.587126 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27d28b89-ed2b-4235-836a-fd70777d8bbd service nova] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:49:41.604469 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 705/1425] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:49:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 374 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:49:42.711499 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa70d677-cbb7-4e4f-953e-f1edf25098fa None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:49:42.712176 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa70d677-cbb7-4e4f-953e-f1edf25098fa None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:49:42.712176 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa70d677-cbb7-4e4f-953e-f1edf25098fa None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:49:42.713406 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 721/1426] 127.0.0.1 () {54 vars in 901 bytes} [Sun Oct 9 00:49:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:49:42.720180 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6daf5eca-1851-444d-ac8e-991450b36642 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:49:42.720633 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6daf5eca-1851-444d-ac8e-991450b36642 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:49:42.720755 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6daf5eca-1851-444d-ac8e-991450b36642 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:49:42.794662 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-6daf5eca-1851-444d-ac8e-991450b36642 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:49:42.795986 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-6daf5eca-1851-444d-ac8e-991450b36642 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:49:42.898179 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 706/1427] 127.0.0.1 () {58 vars in 998 bytes} [Sun Oct 9 00:49:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:49:43.050516 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa70d677-cbb7-4e4f-953e-f1edf25098fa None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:49:43.051998 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-aa70d677-cbb7-4e4f-953e-f1edf25098fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:49:43.136963 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:49:43.198611 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa70d677-cbb7-4e4f-953e-f1edf25098fa None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:49:43.308615 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aea58bf-28e7-4bae-bb6d-a14fe8f73001 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a5d95969e74157bb2d2a57207e1651', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:49:43.309134 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aea58bf-28e7-4bae-bb6d-a14fe8f73001 service nova] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:49:43.309410 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aea58bf-28e7-4bae-bb6d-a14fe8f73001 service nova] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:49:43.309677 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aea58bf-28e7-4bae-bb6d-a14fe8f73001 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:49:43.311331 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-5aea58bf-28e7-4bae-bb6d-a14fe8f73001 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:49:43.359185 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:49:43.400249 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aea58bf-28e7-4bae-bb6d-a14fe8f73001 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:49:43.401850 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aea58bf-28e7-4bae-bb6d-a14fe8f73001 service nova] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:49:43.419302 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 722/1428] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:49:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 369 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:49:44.467285 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d28c93f-c11b-496d-9b89-4e4437864c9e None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:49:44.467867 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d28c93f-c11b-496d-9b89-4e4437864c9e None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:49:44.467867 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d28c93f-c11b-496d-9b89-4e4437864c9e None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:49:44.469170 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 707/1429] 127.0.0.1 () {54 vars in 901 bytes} [Sun Oct 9 00:49:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:49:44.475158 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c770e399-3510-4d01-b1bc-ae3cb1f32b73 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:49:44.475646 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c770e399-3510-4d01-b1bc-ae3cb1f32b73 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:49:44.475762 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c770e399-3510-4d01-b1bc-ae3cb1f32b73 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:49:44.551366 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-c770e399-3510-4d01-b1bc-ae3cb1f32b73 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:49:44.552598 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-c770e399-3510-4d01-b1bc-ae3cb1f32b73 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:49:44.646224 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 723/1430] 127.0.0.1 () {58 vars in 998 bytes} [Sun Oct 9 00:49:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:49:44.794139 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d28c93f-c11b-496d-9b89-4e4437864c9e None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:49:44.796462 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-3d28c93f-c11b-496d-9b89-4e4437864c9e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:49:44.844523 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:49:44.941501 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d28c93f-c11b-496d-9b89-4e4437864c9e None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:49:45.052566 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69ff23ef-7a89-44c2-8b52-dc201aa3a2d2 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a5d95969e74157bb2d2a57207e1651', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:49:45.053087 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ff23ef-7a89-44c2-8b52-dc201aa3a2d2 service nova] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:49:45.053362 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ff23ef-7a89-44c2-8b52-dc201aa3a2d2 service nova] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:49:45.053631 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ff23ef-7a89-44c2-8b52-dc201aa3a2d2 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:49:45.055457 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-69ff23ef-7a89-44c2-8b52-dc201aa3a2d2 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:49:45.062333 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:49:45.146688 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69ff23ef-7a89-44c2-8b52-dc201aa3a2d2 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:49:45.148323 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69ff23ef-7a89-44c2-8b52-dc201aa3a2d2 service nova] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:49:45.166483 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 708/1431] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:49:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 373 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:49:46.217754 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa367263-00c3-4ee7-a8d3-adbe49a10f0d None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:49:46.218325 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa367263-00c3-4ee7-a8d3-adbe49a10f0d None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:49:46.218325 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa367263-00c3-4ee7-a8d3-adbe49a10f0d None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:49:46.219718 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 724/1432] 127.0.0.1 () {54 vars in 901 bytes} [Sun Oct 9 00:49:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:49:46.225833 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ed3205-fb48-4dd3-9411-706f16062804 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:49:46.226252 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ed3205-fb48-4dd3-9411-706f16062804 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:49:46.226369 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ed3205-fb48-4dd3-9411-706f16062804 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:49:46.301667 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-f8ed3205-fb48-4dd3-9411-706f16062804 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:49:46.302876 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-f8ed3205-fb48-4dd3-9411-706f16062804 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:49:46.369422 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:49:46.398113 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 709/1433] 127.0.0.1 () {58 vars in 998 bytes} [Sun Oct 9 00:49:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:49:46.546909 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa367263-00c3-4ee7-a8d3-adbe49a10f0d None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:49:46.548520 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-fa367263-00c3-4ee7-a8d3-adbe49a10f0d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:49:46.566332 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:49:46.693221 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa367263-00c3-4ee7-a8d3-adbe49a10f0d None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:49:46.777969 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:49:46.805385 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e82fe75d-944e-4e71-b6eb-a60b4b967944 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a5d95969e74157bb2d2a57207e1651', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:49:46.805988 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e82fe75d-944e-4e71-b6eb-a60b4b967944 service nova] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:49:46.806268 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e82fe75d-944e-4e71-b6eb-a60b4b967944 service nova] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:49:46.806542 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e82fe75d-944e-4e71-b6eb-a60b4b967944 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:49:46.808147 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-e82fe75d-944e-4e71-b6eb-a60b4b967944 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:49:46.897689 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e82fe75d-944e-4e71-b6eb-a60b4b967944 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:49:46.899275 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e82fe75d-944e-4e71-b6eb-a60b4b967944 service nova] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:49:46.917291 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 725/1434] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:49:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 371 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:49:47.966438 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5891c66f-8f8a-4b13-ba30-69fb4972f5eb None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:49:47.966995 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5891c66f-8f8a-4b13-ba30-69fb4972f5eb None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:49:47.966995 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5891c66f-8f8a-4b13-ba30-69fb4972f5eb None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:49:47.968315 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 710/1435] 127.0.0.1 () {54 vars in 901 bytes} [Sun Oct 9 00:49:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:49:47.974641 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd3c9d94-e039-4bda-9f41-20dc98c39823 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:49:47.975127 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd3c9d94-e039-4bda-9f41-20dc98c39823 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:49:47.975223 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd3c9d94-e039-4bda-9f41-20dc98c39823 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:49:48.050162 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-dd3c9d94-e039-4bda-9f41-20dc98c39823 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:49:48.051362 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-dd3c9d94-e039-4bda-9f41-20dc98c39823 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:49:48.094722 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:49:48.148979 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 726/1436] 127.0.0.1 () {58 vars in 998 bytes} [Sun Oct 9 00:49:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:49:48.299217 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5891c66f-8f8a-4b13-ba30-69fb4972f5eb None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:49:48.300769 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-5891c66f-8f8a-4b13-ba30-69fb4972f5eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:49:48.444691 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5891c66f-8f8a-4b13-ba30-69fb4972f5eb None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:49:48.497305 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:49:48.553963 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09fb7a17-100c-4b9f-9acc-273526237e71 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a5d95969e74157bb2d2a57207e1651', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:49:48.554561 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09fb7a17-100c-4b9f-9acc-273526237e71 service nova] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:49:48.554834 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09fb7a17-100c-4b9f-9acc-273526237e71 service nova] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:49:48.555103 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09fb7a17-100c-4b9f-9acc-273526237e71 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:49:48.556709 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-09fb7a17-100c-4b9f-9acc-273526237e71 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:49:48.643259 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09fb7a17-100c-4b9f-9acc-273526237e71 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:49:48.644860 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09fb7a17-100c-4b9f-9acc-273526237e71 service nova] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:49:48.662352 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 711/1437] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:49:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 364 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:49:49.709213 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f80f6ad1-2dd1-49ff-80a0-d1b629d306c3 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:49:49.709812 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f80f6ad1-2dd1-49ff-80a0-d1b629d306c3 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:49:49.709812 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f80f6ad1-2dd1-49ff-80a0-d1b629d306c3 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:49:49.711078 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 727/1438] 127.0.0.1 () {54 vars in 901 bytes} [Sun Oct 9 00:49:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:49:49.717561 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e23c0e3e-42a5-45ac-8dee-3d129f30bda1 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:49:49.717994 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e23c0e3e-42a5-45ac-8dee-3d129f30bda1 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:49:49.718137 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e23c0e3e-42a5-45ac-8dee-3d129f30bda1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:49:49.792904 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-e23c0e3e-42a5-45ac-8dee-3d129f30bda1 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:49:49.794573 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-e23c0e3e-42a5-45ac-8dee-3d129f30bda1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:49:49.796968 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:49:49.893973 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 712/1439] 127.0.0.1 () {58 vars in 998 bytes} [Sun Oct 9 00:49:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:49:50.044843 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f80f6ad1-2dd1-49ff-80a0-d1b629d306c3 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:49:50.046351 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-f80f6ad1-2dd1-49ff-80a0-d1b629d306c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:49:50.190146 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f80f6ad1-2dd1-49ff-80a0-d1b629d306c3 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:49:50.217195 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:49:50.301922 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90d62748-49f3-414c-92e7-e38812cbf202 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a5d95969e74157bb2d2a57207e1651', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:49:50.302444 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d62748-49f3-414c-92e7-e38812cbf202 service nova] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:49:50.302970 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d62748-49f3-414c-92e7-e38812cbf202 service nova] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:49:50.303242 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d62748-49f3-414c-92e7-e38812cbf202 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:49:50.304863 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-90d62748-49f3-414c-92e7-e38812cbf202 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:49:50.393422 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90d62748-49f3-414c-92e7-e38812cbf202 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:49:50.395022 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90d62748-49f3-414c-92e7-e38812cbf202 service nova] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:49:50.412466 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 728/1440] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:49:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 368 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:49:52.506970 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ec83a4-149f-45db-8d71-da48fb2e4d2e None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:49:52.507514 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ec83a4-149f-45db-8d71-da48fb2e4d2e None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:49:52.507514 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ec83a4-149f-45db-8d71-da48fb2e4d2e None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:49:52.508863 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 713/1441] 127.0.0.1 () {54 vars in 917 bytes} [Sun Oct 9 00:49:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:49:52.515680 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f54f1a9-0672-4f25-a6b9-5182adbc3533 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:49:52.516157 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f54f1a9-0672-4f25-a6b9-5182adbc3533 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:49:52.516755 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f54f1a9-0672-4f25-a6b9-5182adbc3533 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:49:52.559204 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:49:52.587179 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-5f54f1a9-0672-4f25-a6b9-5182adbc3533 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:49:52.588562 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-5f54f1a9-0672-4f25-a6b9-5182adbc3533 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:49:52.688595 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 729/1442] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Oct 9 00:49:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:49:52.975125 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10ec83a4-149f-45db-8d71-da48fb2e4d2e None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:49:52.976697 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-10ec83a4-149f-45db-8d71-da48fb2e4d2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:49:53.077693 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10ec83a4-149f-45db-8d71-da48fb2e4d2e None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:49:53.110278 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:49:53.151137 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf1ea2d3-5663-4cdd-8f4e-0c0f3def3241 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:49:53.152355 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf1ea2d3-5663-4cdd-8f4e-0c0f3def3241 service neutron] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:49:53.152670 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf1ea2d3-5663-4cdd-8f4e-0c0f3def3241 service neutron] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:49:53.152941 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf1ea2d3-5663-4cdd-8f4e-0c0f3def3241 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:49:53.154563 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-cf1ea2d3-5663-4cdd-8f4e-0c0f3def3241 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:49:53.260437 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf1ea2d3-5663-4cdd-8f4e-0c0f3def3241 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:49:53.262414 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf1ea2d3-5663-4cdd-8f4e-0c0f3def3241 service neutron] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:49:53.279746 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 714/1443] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:49:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:49:55.535830 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f9a43c0-28c4-4b96-9145-2328435fb077 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:49:55.536374 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f9a43c0-28c4-4b96-9145-2328435fb077 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:49:55.536505 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f9a43c0-28c4-4b96-9145-2328435fb077 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:49:55.537831 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 730/1444] 127.0.0.1 () {54 vars in 917 bytes} [Sun Oct 9 00:49:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:49:55.542901 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b16a13c3-e522-4ac0-b1b7-34c84f7337be None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:49:55.543347 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b16a13c3-e522-4ac0-b1b7-34c84f7337be None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:49:55.543445 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b16a13c3-e522-4ac0-b1b7-34c84f7337be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:49:55.610413 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-b16a13c3-e522-4ac0-b1b7-34c84f7337be None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:49:55.611644 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-b16a13c3-e522-4ac0-b1b7-34c84f7337be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:49:55.624925 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:49:55.708067 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 715/1445] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Oct 9 00:49:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:49:55.990716 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f9a43c0-28c4-4b96-9145-2328435fb077 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:49:55.992321 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-8f9a43c0-28c4-4b96-9145-2328435fb077 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:49:56.095033 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f9a43c0-28c4-4b96-9145-2328435fb077 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:49:56.108491 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:49:56.171471 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efed43ae-e05f-4767-9ffc-df6b12a43af2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:49:56.172096 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efed43ae-e05f-4767-9ffc-df6b12a43af2 service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:49:56.172376 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efed43ae-e05f-4767-9ffc-df6b12a43af2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:49:56.172648 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efed43ae-e05f-4767-9ffc-df6b12a43af2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:49:56.174235 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-efed43ae-e05f-4767-9ffc-df6b12a43af2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:49:56.283866 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efed43ae-e05f-4767-9ffc-df6b12a43af2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:49:56.285216 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efed43ae-e05f-4767-9ffc-df6b12a43af2 service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:49:56.304279 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 731/1446] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:49:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:49:58.751839 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe1e2313-cacc-4538-82cb-b4d7eec425aa None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:49:58.752421 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe1e2313-cacc-4538-82cb-b4d7eec425aa None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:49:58.752421 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe1e2313-cacc-4538-82cb-b4d7eec425aa None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:49:58.753774 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 716/1447] 127.0.0.1 () {54 vars in 917 bytes} [Sun Oct 9 00:49:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:49:58.758962 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb09ca0-289b-4d38-8117-6c8239181b83 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:49:58.760005 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb09ca0-289b-4d38-8117-6c8239181b83 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:49:58.760124 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb09ca0-289b-4d38-8117-6c8239181b83 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:49:58.807824 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:49:58.826267 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-3eb09ca0-289b-4d38-8117-6c8239181b83 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:49:58.827472 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-3eb09ca0-289b-4d38-8117-6c8239181b83 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:49:58.923304 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 732/1448] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Oct 9 00:49:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:49:59.209134 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe1e2313-cacc-4538-82cb-b4d7eec425aa None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:49:59.210677 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-fe1e2313-cacc-4538-82cb-b4d7eec425aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:49:59.312994 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe1e2313-cacc-4538-82cb-b4d7eec425aa None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:49:59.352528 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:49:59.386257 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56780a57-fb35-4bdd-8b1d-13c2ef47a9d1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:49:59.386971 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56780a57-fb35-4bdd-8b1d-13c2ef47a9d1 service neutron] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:49:59.387263 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56780a57-fb35-4bdd-8b1d-13c2ef47a9d1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:49:59.387557 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56780a57-fb35-4bdd-8b1d-13c2ef47a9d1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:49:59.389205 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-56780a57-fb35-4bdd-8b1d-13c2ef47a9d1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:49:59.496887 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56780a57-fb35-4bdd-8b1d-13c2ef47a9d1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:49:59.498287 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56780a57-fb35-4bdd-8b1d-13c2ef47a9d1 service neutron] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:49:59.516231 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 717/1449] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:49:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:50:01.773273 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa8fd37-0410-4ef5-ba6c-2aeba9dde463 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:50:01.773848 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa8fd37-0410-4ef5-ba6c-2aeba9dde463 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:50:01.773848 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa8fd37-0410-4ef5-ba6c-2aeba9dde463 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:50:01.775731 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 733/1450] 127.0.0.1 () {54 vars in 917 bytes} [Sun Oct 9 00:50:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:50:01.781223 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10f487ce-a183-415d-9995-90ec24c17512 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:50:01.781640 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10f487ce-a183-415d-9995-90ec24c17512 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:50:01.781804 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10f487ce-a183-415d-9995-90ec24c17512 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:50:01.847214 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-10f487ce-a183-415d-9995-90ec24c17512 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:50:01.848447 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-10f487ce-a183-415d-9995-90ec24c17512 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:50:01.869845 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:50:01.946893 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 718/1451] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Oct 9 00:50:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:50:02.230630 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfa8fd37-0410-4ef5-ba6c-2aeba9dde463 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:50:02.232248 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-cfa8fd37-0410-4ef5-ba6c-2aeba9dde463 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:50:02.334533 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfa8fd37-0410-4ef5-ba6c-2aeba9dde463 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:50:02.354499 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:50:02.407519 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77df10f0-522a-4fd6-bc1a-c25b539fce0a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:50:02.408151 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77df10f0-522a-4fd6-bc1a-c25b539fce0a service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:50:02.408432 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77df10f0-522a-4fd6-bc1a-c25b539fce0a service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:50:02.408703 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77df10f0-522a-4fd6-bc1a-c25b539fce0a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:50:02.410300 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-77df10f0-522a-4fd6-bc1a-c25b539fce0a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:50:02.520388 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77df10f0-522a-4fd6-bc1a-c25b539fce0a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:50:02.522217 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77df10f0-522a-4fd6-bc1a-c25b539fce0a service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:50:02.539767 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 734/1452] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:50:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:50:04.742931 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449da65a-7747-4e37-96b3-676841908e8b None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:50:04.743462 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449da65a-7747-4e37-96b3-676841908e8b None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:50:04.743462 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449da65a-7747-4e37-96b3-676841908e8b None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:50:04.744832 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 719/1453] 127.0.0.1 () {54 vars in 917 bytes} [Sun Oct 9 00:50:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:50:04.750052 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae7b27b5-4702-4352-88b4-c2f41e446a4a None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:50:04.750482 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae7b27b5-4702-4352-88b4-c2f41e446a4a None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:50:04.750594 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae7b27b5-4702-4352-88b4-c2f41e446a4a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:50:04.813666 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:50:04.816529 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-ae7b27b5-4702-4352-88b4-c2f41e446a4a None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:50:04.818372 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-ae7b27b5-4702-4352-88b4-c2f41e446a4a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:50:04.914619 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 735/1454] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Oct 9 00:50:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:50:05.202284 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-449da65a-7747-4e37-96b3-676841908e8b None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:50:05.203907 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-449da65a-7747-4e37-96b3-676841908e8b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:50:05.306894 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-449da65a-7747-4e37-96b3-676841908e8b None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:50:05.355642 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:50:05.379757 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c811547-fa18-49de-976d-6f2ef955c84a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:50:05.380354 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c811547-fa18-49de-976d-6f2ef955c84a service neutron] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:50:05.380631 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c811547-fa18-49de-976d-6f2ef955c84a service neutron] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:50:05.380898 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c811547-fa18-49de-976d-6f2ef955c84a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:50:05.383013 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-5c811547-fa18-49de-976d-6f2ef955c84a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:50:05.489640 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c811547-fa18-49de-976d-6f2ef955c84a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:50:05.491065 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c811547-fa18-49de-976d-6f2ef955c84a service neutron] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:50:05.509842 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 720/1455] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:50:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:50:07.708522 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeab4950-d015-4778-87c5-bf894baf051f None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:50:07.709091 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeab4950-d015-4778-87c5-bf894baf051f None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:50:07.709091 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeab4950-d015-4778-87c5-bf894baf051f None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:50:07.710431 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 736/1456] 127.0.0.1 () {54 vars in 917 bytes} [Sun Oct 9 00:50:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:50:07.715741 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-181ab85e-9616-4c98-b23e-a0a7d9120360 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:50:07.716143 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-181ab85e-9616-4c98-b23e-a0a7d9120360 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:50:07.716310 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-181ab85e-9616-4c98-b23e-a0a7d9120360 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:50:07.781983 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-181ab85e-9616-4c98-b23e-a0a7d9120360 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:50:07.783102 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-181ab85e-9616-4c98-b23e-a0a7d9120360 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:50:07.812536 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:50:07.883117 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 721/1457] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Oct 9 00:50:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:50:08.166767 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeab4950-d015-4778-87c5-bf894baf051f None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:50:08.168362 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-eeab4950-d015-4778-87c5-bf894baf051f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:50:08.272028 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeab4950-d015-4778-87c5-bf894baf051f None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:50:08.300505 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:50:08.346251 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d787d940-d4bc-49da-b612-92500e048c93 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:50:08.346778 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d787d940-d4bc-49da-b612-92500e048c93 service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:50:08.347058 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d787d940-d4bc-49da-b612-92500e048c93 service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:50:08.347331 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d787d940-d4bc-49da-b612-92500e048c93 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:50:08.348948 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-d787d940-d4bc-49da-b612-92500e048c93 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:50:08.456258 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d787d940-d4bc-49da-b612-92500e048c93 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:50:08.457679 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d787d940-d4bc-49da-b612-92500e048c93 service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:50:08.476502 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 737/1458] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:50:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:50:10.700070 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e580a0-9afe-4fe5-a1e2-3d1b6475b3f1 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:50:10.700600 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e580a0-9afe-4fe5-a1e2-3d1b6475b3f1 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:50:10.700723 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e580a0-9afe-4fe5-a1e2-3d1b6475b3f1 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:50:10.702027 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 722/1459] 127.0.0.1 () {54 vars in 917 bytes} [Sun Oct 9 00:50:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:50:10.707284 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05e10157-c6d0-40d4-bfaf-5af4c410fbe1 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:50:10.707756 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05e10157-c6d0-40d4-bfaf-5af4c410fbe1 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:50:10.707864 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05e10157-c6d0-40d4-bfaf-5af4c410fbe1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:50:10.772732 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-05e10157-c6d0-40d4-bfaf-5af4c410fbe1 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:50:10.773941 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-05e10157-c6d0-40d4-bfaf-5af4c410fbe1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:50:10.782156 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:50:10.869391 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 738/1460] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Oct 9 00:50:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:50:11.153189 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e580a0-9afe-4fe5-a1e2-3d1b6475b3f1 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:50:11.154789 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-c0e580a0-9afe-4fe5-a1e2-3d1b6475b3f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:50:11.256825 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e580a0-9afe-4fe5-a1e2-3d1b6475b3f1 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:50:11.313486 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:50:11.332005 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-871ca995-0190-4a66-afe4-f72e0a1aba14 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:50:11.332548 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-871ca995-0190-4a66-afe4-f72e0a1aba14 service neutron] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:50:11.332829 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-871ca995-0190-4a66-afe4-f72e0a1aba14 service neutron] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:50:11.333098 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-871ca995-0190-4a66-afe4-f72e0a1aba14 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:50:11.334771 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-871ca995-0190-4a66-afe4-f72e0a1aba14 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:50:11.443790 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-871ca995-0190-4a66-afe4-f72e0a1aba14 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:50:11.445204 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-871ca995-0190-4a66-afe4-f72e0a1aba14 service neutron] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:50:11.463291 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 723/1461] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:50:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:50:13.663913 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b48612e-3e70-46cc-aba3-8557e4e68124 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:50:13.664580 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b48612e-3e70-46cc-aba3-8557e4e68124 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:50:13.664580 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b48612e-3e70-46cc-aba3-8557e4e68124 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:50:13.665801 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 739/1462] 127.0.0.1 () {54 vars in 917 bytes} [Sun Oct 9 00:50:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:50:13.671026 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-929f80c3-b3a5-4876-aa04-aae42cd29016 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:50:13.671457 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-929f80c3-b3a5-4876-aa04-aae42cd29016 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:50:13.671565 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-929f80c3-b3a5-4876-aa04-aae42cd29016 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:50:13.736674 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-929f80c3-b3a5-4876-aa04-aae42cd29016 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:50:13.737881 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-929f80c3-b3a5-4876-aa04-aae42cd29016 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:50:13.775649 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:50:13.835400 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 724/1463] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Oct 9 00:50:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:50:14.124167 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b48612e-3e70-46cc-aba3-8557e4e68124 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:50:14.125741 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-7b48612e-3e70-46cc-aba3-8557e4e68124 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:50:14.227611 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b48612e-3e70-46cc-aba3-8557e4e68124 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:50:14.265051 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:50:14.300544 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da833386-af6d-47ef-ba14-bb041bfaaee4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:50:14.301169 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da833386-af6d-47ef-ba14-bb041bfaaee4 service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:50:14.301451 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da833386-af6d-47ef-ba14-bb041bfaaee4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:50:14.302411 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da833386-af6d-47ef-ba14-bb041bfaaee4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:50:14.304114 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-da833386-af6d-47ef-ba14-bb041bfaaee4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:50:14.410324 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da833386-af6d-47ef-ba14-bb041bfaaee4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:50:14.412045 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da833386-af6d-47ef-ba14-bb041bfaaee4 service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:50:14.430342 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 740/1464] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:50:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:50:16.639715 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea0bd42e-a5a6-42b8-8469-dd4cbae27c74 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:50:16.640264 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea0bd42e-a5a6-42b8-8469-dd4cbae27c74 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:50:16.640264 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea0bd42e-a5a6-42b8-8469-dd4cbae27c74 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:50:16.641576 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 725/1465] 127.0.0.1 () {54 vars in 917 bytes} [Sun Oct 9 00:50:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:50:16.646913 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c98101e7-69c3-43f0-9335-51f11de1a755 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:50:16.647342 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c98101e7-69c3-43f0-9335-51f11de1a755 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:50:16.647476 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c98101e7-69c3-43f0-9335-51f11de1a755 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:50:16.713913 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-c98101e7-69c3-43f0-9335-51f11de1a755 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:50:16.715143 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-c98101e7-69c3-43f0-9335-51f11de1a755 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:50:16.733017 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:50:16.810833 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 741/1466] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Oct 9 00:50:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:50:17.098621 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea0bd42e-a5a6-42b8-8469-dd4cbae27c74 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:50:17.100198 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-ea0bd42e-a5a6-42b8-8469-dd4cbae27c74 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:50:17.203214 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea0bd42e-a5a6-42b8-8469-dd4cbae27c74 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:50:17.269424 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:50:17.276778 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf99fdd3-9431-4991-a0d8-4fff0fb7a5ba service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:50:17.277386 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf99fdd3-9431-4991-a0d8-4fff0fb7a5ba service neutron] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:50:17.277664 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf99fdd3-9431-4991-a0d8-4fff0fb7a5ba service neutron] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:50:17.277931 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf99fdd3-9431-4991-a0d8-4fff0fb7a5ba service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:50:17.279528 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-cf99fdd3-9431-4991-a0d8-4fff0fb7a5ba service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:50:17.385780 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf99fdd3-9431-4991-a0d8-4fff0fb7a5ba service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:50:17.387131 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf99fdd3-9431-4991-a0d8-4fff0fb7a5ba service neutron] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:50:17.404545 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 726/1467] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:50:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:50:19.564406 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c1856c-16c7-46b7-9b95-ef8aae87914f None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:50:19.564971 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c1856c-16c7-46b7-9b95-ef8aae87914f None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:50:19.564971 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c1856c-16c7-46b7-9b95-ef8aae87914f None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:50:19.566851 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 742/1468] 127.0.0.1 () {54 vars in 917 bytes} [Sun Oct 9 00:50:19 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:50:19.572100 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e0e950e-05ce-4a4e-8e72-3c7a35392d15 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:50:19.572484 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e0e950e-05ce-4a4e-8e72-3c7a35392d15 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:50:19.572650 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e0e950e-05ce-4a4e-8e72-3c7a35392d15 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:50:19.637997 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-8e0e950e-05ce-4a4e-8e72-3c7a35392d15 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:50:19.639208 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-8e0e950e-05ce-4a4e-8e72-3c7a35392d15 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:50:19.686701 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:50:19.735167 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 727/1469] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Oct 9 00:50:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:50:20.022576 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c1856c-16c7-46b7-9b95-ef8aae87914f None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:50:20.024185 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-14c1856c-16c7-46b7-9b95-ef8aae87914f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:50:20.128629 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c1856c-16c7-46b7-9b95-ef8aae87914f None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:50:20.176544 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:50:20.202232 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa0a11d7-7d90-4082-b1fe-3edc8d96e53a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:50:20.202832 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0a11d7-7d90-4082-b1fe-3edc8d96e53a service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:50:20.203110 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0a11d7-7d90-4082-b1fe-3edc8d96e53a service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:50:20.203379 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0a11d7-7d90-4082-b1fe-3edc8d96e53a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:50:20.205010 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-fa0a11d7-7d90-4082-b1fe-3edc8d96e53a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:50:20.312289 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa0a11d7-7d90-4082-b1fe-3edc8d96e53a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:50:20.313957 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa0a11d7-7d90-4082-b1fe-3edc8d96e53a service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:50:20.332045 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 743/1470] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:50:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:50:29.733605 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=22 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:50:30.329757 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=66 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:50:40.096372 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-688362be-f6f1-4e48-a65c-94b91f2a47c5 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:50:40.096852 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-688362be-f6f1-4e48-a65c-94b91f2a47c5 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:50:40.096963 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-688362be-f6f1-4e48-a65c-94b91f2a47c5 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:50:40.098225 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 728/1471] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:50:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:50:40.103411 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955ae6e0-b5df-43a2-80c3-2c56b548a299 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:50:40.103851 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955ae6e0-b5df-43a2-80c3-2c56b548a299 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:50:40.104001 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955ae6e0-b5df-43a2-80c3-2c56b548a299 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:50:40.170482 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-955ae6e0-b5df-43a2-80c3-2c56b548a299 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:50:40.172333 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-955ae6e0-b5df-43a2-80c3-2c56b548a299 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:50:40.385263 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 744/1472] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:50:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 283 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:50:40.683981 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15dd8895-2c0e-47fd-a472-2a526501c6c3 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:50:40.684293 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15dd8895-2c0e-47fd-a472-2a526501c6c3 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:50:40.684571 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15dd8895-2c0e-47fd-a472-2a526501c6c3 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:50:40.685893 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 729/1473] 10.222.0.33 () {54 vars in 962 bytes} [Sun Oct 9 00:50:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:50:40.691273 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee5b651-4db6-4ff8-a874-c61bbeba02ad None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:50:40.691583 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee5b651-4db6-4ff8-a874-c61bbeba02ad None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:50:40.691887 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee5b651-4db6-4ff8-a874-c61bbeba02ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:50:40.765426 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-5ee5b651-4db6-4ff8-a874-c61bbeba02ad None None] MFA Rules not processed for user `1f7a88094d73471787cc0a287adbd394`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:50:40.766656 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-5ee5b651-4db6-4ff8-a874-c61bbeba02ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:50:40.786441 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:50:40.844003 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 745/1474] 10.222.0.33 () {58 vars in 1059 bytes} [Sun Oct 9 00:50:40 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:50:40.851673 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15dd8895-2c0e-47fd-a472-2a526501c6c3 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:50:40.853247 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-15dd8895-2c0e-47fd-a472-2a526501c6c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:50:40.955283 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15dd8895-2c0e-47fd-a472-2a526501c6c3 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:50:41.027848 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7585e167-ea4e-444c-ba88-1e398e530e02 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1f7a88094d73471787cc0a287adbd394', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:50:41.028461 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7585e167-ea4e-444c-ba88-1e398e530e02 service glance] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:50:41.028743 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7585e167-ea4e-444c-ba88-1e398e530e02 service glance] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:50:41.029015 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7585e167-ea4e-444c-ba88-1e398e530e02 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:50:41.031114 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-7585e167-ea4e-444c-ba88-1e398e530e02 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:50:41.084723 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:50:41.140002 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7585e167-ea4e-444c-ba88-1e398e530e02 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:50:41.142188 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7585e167-ea4e-444c-ba88-1e398e530e02 service glance] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:50:41.161158 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 730/1475] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:50:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 09 00:50:43.383530 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da76ffc-9d00-4ed7-9e78-d6f6abd26d55 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:50:43.384138 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da76ffc-9d00-4ed7-9e78-d6f6abd26d55 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:50:43.384138 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da76ffc-9d00-4ed7-9e78-d6f6abd26d55 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:50:43.385386 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 746/1476] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:50:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:50:43.391141 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c86a31ce-2092-47c3-ab17-d9c2ed83d6c6 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:50:43.391558 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c86a31ce-2092-47c3-ab17-d9c2ed83d6c6 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:50:43.391713 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c86a31ce-2092-47c3-ab17-d9c2ed83d6c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:50:43.456419 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-c86a31ce-2092-47c3-ab17-d9c2ed83d6c6 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:50:43.457627 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-c86a31ce-2092-47c3-ab17-d9c2ed83d6c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:50:43.551330 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 731/1477] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:50:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:50:43.827144 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0da76ffc-9d00-4ed7-9e78-d6f6abd26d55 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:50:43.828710 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-0da76ffc-9d00-4ed7-9e78-d6f6abd26d55 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:50:43.951388 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0da76ffc-9d00-4ed7-9e78-d6f6abd26d55 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:50:44.003960 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:50:44.048628 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c472ec7-1cdb-4942-a090-56cc18525d96 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:50:44.049216 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c472ec7-1cdb-4942-a090-56cc18525d96 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:50:44.049490 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c472ec7-1cdb-4942-a090-56cc18525d96 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:50:44.049754 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c472ec7-1cdb-4942-a090-56cc18525d96 admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:50:44.050379 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c472ec7-1cdb-4942-a090-56cc18525d96 admin admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:50:44.052120 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: WARNING py.warnings [None req-6c472ec7-1cdb-4942-a090-56cc18525d96 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_projects": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 09 00:50:44.052120 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: warnings.warn(msg) Oct 09 00:50:44.052120 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: Oct 09 00:50:44.053243 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c472ec7-1cdb-4942-a090-56cc18525d96 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:50:44.071813 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 747/1478] 10.222.0.33 () {56 vars in 1108 bytes} [Sun Oct 9 00:50:43 2022] GET /identity/v3/projects => generated 2553 bytes in 245 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:50:46.211725 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9680b1b0-123b-4cab-a59a-f9c4efe2a47f None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:50:46.212359 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9680b1b0-123b-4cab-a59a-f9c4efe2a47f None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:50:46.212359 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9680b1b0-123b-4cab-a59a-f9c4efe2a47f None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:50:46.213592 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 732/1479] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:50:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:50:46.219231 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d91cf40-7a40-4dcc-a105-dd6dec37a368 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:50:46.219688 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d91cf40-7a40-4dcc-a105-dd6dec37a368 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:50:46.219801 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d91cf40-7a40-4dcc-a105-dd6dec37a368 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:50:46.284185 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-8d91cf40-7a40-4dcc-a105-dd6dec37a368 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:50:46.285391 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-8d91cf40-7a40-4dcc-a105-dd6dec37a368 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:50:46.379368 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 748/1480] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:50:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:50:46.675219 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9680b1b0-123b-4cab-a59a-f9c4efe2a47f None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:50:46.676847 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-9680b1b0-123b-4cab-a59a-f9c4efe2a47f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:50:46.686410 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:50:46.821874 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9680b1b0-123b-4cab-a59a-f9c4efe2a47f None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:50:46.902335 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:50:46.934593 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-942037b8-8ccd-414e-8e27-ce33ec076c67 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a5d95969e74157bb2d2a57207e1651', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:50:46.935676 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-942037b8-8ccd-414e-8e27-ce33ec076c67 service nova] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:50:46.935962 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-942037b8-8ccd-414e-8e27-ce33ec076c67 service nova] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:50:46.936231 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-942037b8-8ccd-414e-8e27-ce33ec076c67 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:50:46.937835 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-942037b8-8ccd-414e-8e27-ce33ec076c67 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:50:47.025493 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-942037b8-8ccd-414e-8e27-ce33ec076c67 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:50:47.027081 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-942037b8-8ccd-414e-8e27-ce33ec076c67 service nova] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:50:47.045349 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 733/1481] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:50:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 371 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:50:49.337245 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870bf438-f339-498d-bd31-fa27ad560a63 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:50:49.337768 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870bf438-f339-498d-bd31-fa27ad560a63 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:50:49.337768 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870bf438-f339-498d-bd31-fa27ad560a63 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:50:49.339062 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 749/1482] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:50:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:50:49.345000 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ac7e80-5cdd-4b98-9ba1-dd3d392fef5b None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:50:49.345439 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ac7e80-5cdd-4b98-9ba1-dd3d392fef5b None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:50:49.345540 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ac7e80-5cdd-4b98-9ba1-dd3d392fef5b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:50:49.409770 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-47ac7e80-5cdd-4b98-9ba1-dd3d392fef5b None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:50:49.410969 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-47ac7e80-5cdd-4b98-9ba1-dd3d392fef5b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:50:49.456618 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:50:49.505743 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 734/1483] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:50:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:50:49.792820 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-870bf438-f339-498d-bd31-fa27ad560a63 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:50:49.794551 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-870bf438-f339-498d-bd31-fa27ad560a63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:50:49.818452 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:50:49.898155 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-870bf438-f339-498d-bd31-fa27ad560a63 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:50:49.970441 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68c005a2-3293-4728-8eee-9057af52489e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:50:49.971036 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c005a2-3293-4728-8eee-9057af52489e service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:50:49.971312 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c005a2-3293-4728-8eee-9057af52489e service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:50:49.971584 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c005a2-3293-4728-8eee-9057af52489e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:50:49.973191 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-68c005a2-3293-4728-8eee-9057af52489e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:50:50.051177 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:50:50.086811 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68c005a2-3293-4728-8eee-9057af52489e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:50:50.088304 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68c005a2-3293-4728-8eee-9057af52489e service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:50:50.106211 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 750/1484] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:50:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 09 00:50:52.299800 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d0fb41-44c9-447d-9374-2f8499533165 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:50:52.300358 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d0fb41-44c9-447d-9374-2f8499533165 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:50:52.300358 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d0fb41-44c9-447d-9374-2f8499533165 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:50:52.301706 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 735/1485] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:50:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:50:52.306984 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a33f017a-a925-41a8-8f58-6c8ecdf3528d None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:50:52.307363 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a33f017a-a925-41a8-8f58-6c8ecdf3528d None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:50:52.307535 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a33f017a-a925-41a8-8f58-6c8ecdf3528d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:50:52.372049 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-a33f017a-a925-41a8-8f58-6c8ecdf3528d None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:50:52.373224 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-a33f017a-a925-41a8-8f58-6c8ecdf3528d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:50:52.468125 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 751/1486] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:50:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:50:52.753427 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67d0fb41-44c9-447d-9374-2f8499533165 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:50:52.755011 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-67d0fb41-44c9-447d-9374-2f8499533165 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:50:52.857565 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67d0fb41-44c9-447d-9374-2f8499533165 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:50:53.046289 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:50:53.047851 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-590e1af5-44ee-4cff-8965-be9050fac499 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:50:53.048464 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-590e1af5-44ee-4cff-8965-be9050fac499 service neutron] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:50:53.048986 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-590e1af5-44ee-4cff-8965-be9050fac499 service neutron] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:50:53.049437 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-590e1af5-44ee-4cff-8965-be9050fac499 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:50:53.051240 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-590e1af5-44ee-4cff-8965-be9050fac499 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:50:53.160188 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-590e1af5-44ee-4cff-8965-be9050fac499 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:50:53.161607 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-590e1af5-44ee-4cff-8965-be9050fac499 service neutron] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:50:53.179393 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 736/1487] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:50:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 426 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:50:55.450127 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d18308e3-3036-4b63-87cc-2ed1fca577e6 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:50:55.450673 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d18308e3-3036-4b63-87cc-2ed1fca577e6 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:50:55.450673 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d18308e3-3036-4b63-87cc-2ed1fca577e6 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:50:55.452050 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 752/1488] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:50:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:50:55.457234 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f13f5c4f-3f14-4633-a334-1a05548d6731 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:50:55.457646 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f13f5c4f-3f14-4633-a334-1a05548d6731 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:50:55.457760 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f13f5c4f-3f14-4633-a334-1a05548d6731 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:50:55.523277 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-f13f5c4f-3f14-4633-a334-1a05548d6731 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:50:55.524508 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-f13f5c4f-3f14-4633-a334-1a05548d6731 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:50:55.576841 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:50:55.620663 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 737/1489] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:50:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:50:55.913214 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d18308e3-3036-4b63-87cc-2ed1fca577e6 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:50:55.915060 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-d18308e3-3036-4b63-87cc-2ed1fca577e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:50:55.951164 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:50:56.017595 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d18308e3-3036-4b63-87cc-2ed1fca577e6 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:50:56.094747 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ae1b61d-4c44-412f-99fd-2fa5e5236a19 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:50:56.095341 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae1b61d-4c44-412f-99fd-2fa5e5236a19 service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:50:56.095644 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae1b61d-4c44-412f-99fd-2fa5e5236a19 service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:50:56.095917 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae1b61d-4c44-412f-99fd-2fa5e5236a19 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:50:56.097472 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-7ae1b61d-4c44-412f-99fd-2fa5e5236a19 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:50:56.178281 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:50:56.207341 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ae1b61d-4c44-412f-99fd-2fa5e5236a19 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:50:56.208816 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ae1b61d-4c44-412f-99fd-2fa5e5236a19 service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:50:56.227366 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 753/1490] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:50:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 315 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:50:58.577009 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad5d0e5d-f085-46f3-ba14-feedf21f2920 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:50:58.577434 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad5d0e5d-f085-46f3-ba14-feedf21f2920 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:50:58.577550 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad5d0e5d-f085-46f3-ba14-feedf21f2920 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:50:58.578876 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 738/1491] 10.222.0.33 () {54 vars in 927 bytes} [Sun Oct 9 00:50:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:50:58.583991 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82dd1662-2970-47d1-b4fd-04faccad39cc None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:50:58.584429 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82dd1662-2970-47d1-b4fd-04faccad39cc None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:50:58.584557 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82dd1662-2970-47d1-b4fd-04faccad39cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:50:58.648969 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-82dd1662-2970-47d1-b4fd-04faccad39cc None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:50:58.650161 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-82dd1662-2970-47d1-b4fd-04faccad39cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:50:58.743779 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 754/1492] 10.222.0.33 () {58 vars in 1024 bytes} [Sun Oct 9 00:50:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:50:59.036400 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad5d0e5d-f085-46f3-ba14-feedf21f2920 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:50:59.037951 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-ad5d0e5d-f085-46f3-ba14-feedf21f2920 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:50:59.141231 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad5d0e5d-f085-46f3-ba14-feedf21f2920 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:50:59.213399 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f65e5d56-fce2-420a-935e-ac8d4088cdd3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:50:59.213996 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f65e5d56-fce2-420a-935e-ac8d4088cdd3 service neutron] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:50:59.214297 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f65e5d56-fce2-420a-935e-ac8d4088cdd3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:50:59.214564 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f65e5d56-fce2-420a-935e-ac8d4088cdd3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:50:59.216175 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-f65e5d56-fce2-420a-935e-ac8d4088cdd3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:50:59.224432 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:50:59.323732 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f65e5d56-fce2-420a-935e-ac8d4088cdd3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:50:59.325155 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f65e5d56-fce2-420a-935e-ac8d4088cdd3 service neutron] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:50:59.342440 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 739/1493] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:50:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:51:08.742256 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=82 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:51:09.340346 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=50 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:52:18.669114 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d200de54-153f-42db-b311-582654e149d1 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:52:18.669611 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d200de54-153f-42db-b311-582654e149d1 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:52:18.669729 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d200de54-153f-42db-b311-582654e149d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:52:18.744053 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-d200de54-153f-42db-b311-582654e149d1 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:52:18.745244 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-d200de54-153f-42db-b311-582654e149d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:52:18.840553 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 755/1494] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:52:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:52:18.848313 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-348fe8c6-1a99-4820-b798-28b33f6ad3da None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:52:18.848812 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-348fe8c6-1a99-4820-b798-28b33f6ad3da None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:52:18.848965 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-348fe8c6-1a99-4820-b798-28b33f6ad3da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:52:18.926803 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-348fe8c6-1a99-4820-b798-28b33f6ad3da None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:52:18.928117 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-348fe8c6-1a99-4820-b798-28b33f6ad3da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:52:19.033934 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 740/1495] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:52:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:52:19.038487 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d200de54-153f-42db-b311-582654e149d1 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:52:19.039775 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-d200de54-153f-42db-b311-582654e149d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:52:19.051465 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d200de54-153f-42db-b311-582654e149d1 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:52:19.113263 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:52:19.147450 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a52d993-b608-458c-98dd-328ce1234999 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:52:19.148074 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a52d993-b608-458c-98dd-328ce1234999 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:52:19.148342 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a52d993-b608-458c-98dd-328ce1234999 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:52:19.148606 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a52d993-b608-458c-98dd-328ce1234999 admin admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:52:19.149229 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a52d993-b608-458c-98dd-328ce1234999 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:52:19.149522 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a52d993-b608-458c-98dd-328ce1234999 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:52:19.150514 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: WARNING py.warnings [None req-6a52d993-b608-458c-98dd-328ce1234999 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 09 00:52:19.150514 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: warnings.warn(msg) Oct 09 00:52:19.150514 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: Oct 09 00:52:19.151380 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a52d993-b608-458c-98dd-328ce1234999 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:52:19.162757 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 756/1496] 10.222.0.33 () {58 vars in 1153 bytes} [Sun Oct 9 00:52:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 125 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:52:19.171076 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-348fe8c6-1a99-4820-b798-28b33f6ad3da None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:52:19.180519 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-348fe8c6-1a99-4820-b798-28b33f6ad3da None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:52:19.241022 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:52:19.274087 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-122205d9-59ed-4280-8050-93301026c80a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:52:19.274677 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-122205d9-59ed-4280-8050-93301026c80a admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:52:19.274949 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-122205d9-59ed-4280-8050-93301026c80a admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:52:19.275213 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-122205d9-59ed-4280-8050-93301026c80a admin admin] PATH_INFO: `/v3/projects` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:52:19.275854 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-122205d9-59ed-4280-8050-93301026c80a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:52:19.277561 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: WARNING py.warnings [None req-122205d9-59ed-4280-8050-93301026c80a admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 09 00:52:19.277561 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: warnings.warn(msg) Oct 09 00:52:19.277561 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: Oct 09 00:52:19.278777 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-122205d9-59ed-4280-8050-93301026c80a admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:52:19.334343 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 741/1497] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 00:52:19 2022] POST /identity/v3/projects => generated 379 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:52:19.338497 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a52d993-b608-458c-98dd-328ce1234999 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:52:19.340017 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-6a52d993-b608-458c-98dd-328ce1234999 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:52:19.372838 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a52d993-b608-458c-98dd-328ce1234999 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:52:19.467611 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7bf3390-1328-4a51-9b43-e6125b153cb0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:52:19.468203 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7bf3390-1328-4a51-9b43-e6125b153cb0 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:52:19.468473 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7bf3390-1328-4a51-9b43-e6125b153cb0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:52:19.468735 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7bf3390-1328-4a51-9b43-e6125b153cb0 admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:52:19.469355 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7bf3390-1328-4a51-9b43-e6125b153cb0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:52:19.470353 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: WARNING py.warnings [None req-d7bf3390-1328-4a51-9b43-e6125b153cb0 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 09 00:52:19.470353 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: warnings.warn(msg) Oct 09 00:52:19.470353 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: Oct 09 00:52:19.472510 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7bf3390-1328-4a51-9b43-e6125b153cb0 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:52:19.513857 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 757/1498] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:52:19 2022] POST /identity/v3/users => generated 290 bytes in 176 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:52:19.517943 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-122205d9-59ed-4280-8050-93301026c80a admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:52:19.519413 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-122205d9-59ed-4280-8050-93301026c80a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:52:19.580975 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-122205d9-59ed-4280-8050-93301026c80a admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:52:19.663185 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:52:19.674383 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e93f58ff-7164-48e4-af2a-269b73ab0436 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:52:19.674990 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e93f58ff-7164-48e4-af2a-269b73ab0436 admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:52:19.675269 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e93f58ff-7164-48e4-af2a-269b73ab0436 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:52:19.675541 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e93f58ff-7164-48e4-af2a-269b73ab0436 admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:52:19.676221 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e93f58ff-7164-48e4-af2a-269b73ab0436 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:52:19.677220 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: WARNING py.warnings [None req-e93f58ff-7164-48e4-af2a-269b73ab0436 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 09 00:52:19.677220 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: warnings.warn(msg) Oct 09 00:52:19.677220 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: Oct 09 00:52:19.678100 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e93f58ff-7164-48e4-af2a-269b73ab0436 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:52:19.690303 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 742/1499] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:52:19 2022] GET /identity/v3/roles => generated 1420 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:52:19.694418 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7bf3390-1328-4a51-9b43-e6125b153cb0 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:52:19.695958 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-d7bf3390-1328-4a51-9b43-e6125b153cb0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:52:19.707072 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7bf3390-1328-4a51-9b43-e6125b153cb0 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:52:19.731645 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:52:19.801986 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76798a53-557e-4f56-b2f3-bc54a1749c8c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:52:19.802605 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76798a53-557e-4f56-b2f3-bc54a1749c8c admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:52:19.802877 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76798a53-557e-4f56-b2f3-bc54a1749c8c admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:52:19.803143 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76798a53-557e-4f56-b2f3-bc54a1749c8c admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:52:19.803833 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76798a53-557e-4f56-b2f3-bc54a1749c8c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:52:19.804873 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: WARNING py.warnings [None req-76798a53-557e-4f56-b2f3-bc54a1749c8c admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 09 00:52:19.804873 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: warnings.warn(msg) Oct 09 00:52:19.804873 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: Oct 09 00:52:19.805976 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76798a53-557e-4f56-b2f3-bc54a1749c8c admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:52:19.818505 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 758/1500] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:52:19 2022] GET /identity/v3/roles => generated 1420 bytes in 125 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:52:19.822783 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e93f58ff-7164-48e4-af2a-269b73ab0436 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:52:19.824307 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-e93f58ff-7164-48e4-af2a-269b73ab0436 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:52:19.857793 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e93f58ff-7164-48e4-af2a-269b73ab0436 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:52:19.951175 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49713e19-5d87-4b34-89ac-0396243a7d2c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:52:19.951778 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49713e19-5d87-4b34-89ac-0396243a7d2c admin admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:52:19.952060 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49713e19-5d87-4b34-89ac-0396243a7d2c admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:52:19.952331 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49713e19-5d87-4b34-89ac-0396243a7d2c admin admin] PATH_INFO: `/v3/projects/96f6edbeffac4880b3ef70141f0668e7/users/0e3d2f4835584bf79c56c1489f52decf/roles/588016933bab436d8b7c3ac76827356d` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:52:19.975185 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49713e19-5d87-4b34-89ac-0396243a7d2c admin admin] RBAC: Authorizing `identity:create_grant(project_id=96f6edbeffac4880b3ef70141f0668e7, user_id=0e3d2f4835584bf79c56c1489f52decf, role_id=588016933bab436d8b7c3ac76827356d)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:52:19.976243 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: WARNING py.warnings [None req-49713e19-5d87-4b34-89ac-0396243a7d2c admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 09 00:52:19.976243 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: warnings.warn(msg) Oct 09 00:52:19.976243 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: Oct 09 00:52:19.978188 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49713e19-5d87-4b34-89ac-0396243a7d2c admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:52:20.001191 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 743/1501] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:52:19 2022] PUT /identity/v3/projects/96f6edbeffac4880b3ef70141f0668e7/users/0e3d2f4835584bf79c56c1489f52decf/roles/588016933bab436d8b7c3ac76827356d => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:52:20.013759 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed152841-66ec-46d9-9e82-e63c21b8707c None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:52:20.014094 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed152841-66ec-46d9-9e82-e63c21b8707c None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:52:20.014374 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed152841-66ec-46d9-9e82-e63c21b8707c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:52:20.069573 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-ed152841-66ec-46d9-9e82-e63c21b8707c None None] MFA Rules not processed for user `0e3d2f4835584bf79c56c1489f52decf`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:52:20.070754 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-ed152841-66ec-46d9-9e82-e63c21b8707c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:52:20.148963 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 759/1502] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:52:20 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:52:20.162910 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b81956f8-82f9-4566-8d11-b1837da21e26 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:52:20.163196 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b81956f8-82f9-4566-8d11-b1837da21e26 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:52:20.163470 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b81956f8-82f9-4566-8d11-b1837da21e26 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:52:20.164824 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 744/1503] 10.222.0.33 () {54 vars in 963 bytes} [Sun Oct 9 00:52:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:52:20.169543 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc69eb7d-1bbe-4a88-8f44-2c09fabeef9c None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:52:20.169828 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc69eb7d-1bbe-4a88-8f44-2c09fabeef9c None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:52:20.170099 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc69eb7d-1bbe-4a88-8f44-2c09fabeef9c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:52:20.201127 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:52:20.245944 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-fc69eb7d-1bbe-4a88-8f44-2c09fabeef9c None None] MFA Rules not processed for user `f3f48335668a4e8f934ac11ff66cd308`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:52:20.247066 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-fc69eb7d-1bbe-4a88-8f44-2c09fabeef9c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:52:20.321480 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 760/1504] 10.222.0.33 () {58 vars in 1060 bytes} [Sun Oct 9 00:52:20 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:52:20.329358 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b81956f8-82f9-4566-8d11-b1837da21e26 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:52:20.331465 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-b81956f8-82f9-4566-8d11-b1837da21e26 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:52:20.380473 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:52:20.436963 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b81956f8-82f9-4566-8d11-b1837da21e26 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:52:20.510156 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc7fc59f-ffd1-40bb-9eda-e843cf7b6fd6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3f48335668a4e8f934ac11ff66cd308', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:52:20.510688 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc7fc59f-ffd1-40bb-9eda-e843cf7b6fd6 service cinder] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:52:20.510972 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc7fc59f-ffd1-40bb-9eda-e843cf7b6fd6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:52:20.511244 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc7fc59f-ffd1-40bb-9eda-e843cf7b6fd6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:52:20.512895 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-cc7fc59f-ffd1-40bb-9eda-e843cf7b6fd6 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:52:20.609403 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc7fc59f-ffd1-40bb-9eda-e843cf7b6fd6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:52:20.610886 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc7fc59f-ffd1-40bb-9eda-e843cf7b6fd6 service cinder] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:52:20.614800 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:52:20.629724 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 745/1505] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:52:20 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 301 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:52:20.674504 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be927fb5-bf7c-47c8-95e4-2061936dce1c None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:52:20.674795 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be927fb5-bf7c-47c8-95e4-2061936dce1c None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:52:20.675069 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be927fb5-bf7c-47c8-95e4-2061936dce1c None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:52:20.676363 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 761/1506] 10.222.0.33 () {52 vars in 830 bytes} [Sun Oct 9 00:52:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:52:20.679873 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc7fc59f-ffd1-40bb-9eda-e843cf7b6fd6 service cinder] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:52:20.681097 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-cc7fc59f-ffd1-40bb-9eda-e843cf7b6fd6 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:52:20.777291 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc7fc59f-ffd1-40bb-9eda-e843cf7b6fd6 service cinder] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:52:20.869200 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06776861-f033-401c-992c-b000cba24936 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:52:20.869832 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06776861-f033-401c-992c-b000cba24936 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:52:20.870075 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06776861-f033-401c-992c-b000cba24936 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:52:20.870341 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06776861-f033-401c-992c-b000cba24936 admin admin] PATH_INFO: `/v3/users/0e3d2f4835584bf79c56c1489f52decf` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:52:20.883258 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06776861-f033-401c-992c-b000cba24936 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0e3d2f4835584bf79c56c1489f52decf)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:52:20.884311 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: WARNING py.warnings [None req-06776861-f033-401c-992c-b000cba24936 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 09 00:52:20.884311 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: warnings.warn(msg) Oct 09 00:52:20.884311 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: Oct 09 00:52:20.885524 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06776861-f033-401c-992c-b000cba24936 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:52:20.894861 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:52:20.950009 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-06776861-f033-401c-992c-b000cba24936 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0e3d2f4835584bf79c56c1489f52decf'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:52:20.965190 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-06776861-f033-401c-992c-b000cba24936 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0e3d2f4835584bf79c56c1489f52decf'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:52:20.965632 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-06776861-f033-401c-992c-b000cba24936 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0e3d2f4835584bf79c56c1489f52decf'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:52:20.974848 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-06776861-f033-401c-992c-b000cba24936 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0e3d2f4835584bf79c56c1489f52decf'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:52:20.997975 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-06776861-f033-401c-992c-b000cba24936 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0e3d2f4835584bf79c56c1489f52decf'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:52:21.003429 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 746/1507] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 00:52:20 2022] DELETE /identity/v3/users/0e3d2f4835584bf79c56c1489f52decf => generated 0 bytes in 324 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:52:21.007766 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ab5946e-c9a5-4933-9a5d-2896cf6880ee None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:52:21.008231 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ab5946e-c9a5-4933-9a5d-2896cf6880ee None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:52:21.008331 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ab5946e-c9a5-4933-9a5d-2896cf6880ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:52:21.064760 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-1ab5946e-c9a5-4933-9a5d-2896cf6880ee None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:52:21.065882 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-1ab5946e-c9a5-4933-9a5d-2896cf6880ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:52:21.140796 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:52:21.155213 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 762/1508] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:52:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:52:21.165706 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06776861-f033-401c-992c-b000cba24936 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:52:21.166963 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-06776861-f033-401c-992c-b000cba24936 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:52:21.264653 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06776861-f033-401c-992c-b000cba24936 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:52:21.319679 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:52:21.340697 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bee859c9-fc93-49c3-a13e-fb7935c2d89f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:52:21.341324 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee859c9-fc93-49c3-a13e-fb7935c2d89f service neutron] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:52:21.341602 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee859c9-fc93-49c3-a13e-fb7935c2d89f service neutron] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:52:21.341893 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee859c9-fc93-49c3-a13e-fb7935c2d89f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:52:21.343461 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-bee859c9-fc93-49c3-a13e-fb7935c2d89f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:52:21.449611 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bee859c9-fc93-49c3-a13e-fb7935c2d89f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:52:21.451038 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bee859c9-fc93-49c3-a13e-fb7935c2d89f service neutron] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:52:21.469376 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 747/1509] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:52:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 304 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:52:21.861155 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ab5946e-c9a5-4933-9a5d-2896cf6880ee None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:52:21.862406 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-1ab5946e-c9a5-4933-9a5d-2896cf6880ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:52:21.874124 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ab5946e-c9a5-4933-9a5d-2896cf6880ee None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:52:21.976189 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff2a7646-de63-4b67-8a65-fb7c619dcaad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:52:21.976815 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2a7646-de63-4b67-8a65-fb7c619dcaad admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:52:21.977098 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2a7646-de63-4b67-8a65-fb7c619dcaad admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:52:21.977366 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2a7646-de63-4b67-8a65-fb7c619dcaad admin admin] PATH_INFO: `/v3/projects/96f6edbeffac4880b3ef70141f0668e7` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:52:21.987260 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff2a7646-de63-4b67-8a65-fb7c619dcaad admin admin] RBAC: Authorizing `identity:delete_project(project_id=96f6edbeffac4880b3ef70141f0668e7)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:52:21.988313 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: WARNING py.warnings [None req-ff2a7646-de63-4b67-8a65-fb7c619dcaad admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 09 00:52:21.988313 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: warnings.warn(msg) Oct 09 00:52:21.988313 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: Oct 09 00:52:21.989506 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff2a7646-de63-4b67-8a65-fb7c619dcaad admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:52:22.045718 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-ff2a7646-de63-4b67-8a65-fb7c619dcaad admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '96f6edbeffac4880b3ef70141f0668e7', 'request_id': 'req-ff2a7646-de63-4b67-8a65-fb7c619dcaad'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:52:22.052037 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-ff2a7646-de63-4b67-8a65-fb7c619dcaad admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '96f6edbeffac4880b3ef70141f0668e7', 'request_id': 'req-ff2a7646-de63-4b67-8a65-fb7c619dcaad'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:52:22.066245 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-ff2a7646-de63-4b67-8a65-fb7c619dcaad admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '96f6edbeffac4880b3ef70141f0668e7', 'request_id': 'req-ff2a7646-de63-4b67-8a65-fb7c619dcaad'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:52:22.093681 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-ff2a7646-de63-4b67-8a65-fb7c619dcaad admin admin] The token cache is being invalidate because project 96f6edbeffac4880b3ef70141f0668e7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 00:52:22.093974 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-ff2a7646-de63-4b67-8a65-fb7c619dcaad admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:52:22.094493 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-ff2a7646-de63-4b67-8a65-fb7c619dcaad admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:52:22.096001 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 763/1510] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 00:52:21 2022] DELETE /identity/v3/projects/96f6edbeffac4880b3ef70141f0668e7 => generated 0 bytes in 236 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:52:31.467106 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=62 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:52:32.091635 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=84,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:53:51.425405 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c7bcf6-342e-4664-9ae5-47d37160c493 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:53:51.426010 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c7bcf6-342e-4664-9ae5-47d37160c493 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:53:51.426010 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c7bcf6-342e-4664-9ae5-47d37160c493 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:53:51.427283 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 748/1511] 127.0.0.1 () {54 vars in 901 bytes} [Sun Oct 9 00:53:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:53:51.432040 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ead434d-2c55-4469-88ce-9f9b3f9097ed None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:53:51.432468 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ead434d-2c55-4469-88ce-9f9b3f9097ed None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:53:51.432573 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ead434d-2c55-4469-88ce-9f9b3f9097ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:53:51.511910 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-7ead434d-2c55-4469-88ce-9f9b3f9097ed None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:53:51.513107 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-7ead434d-2c55-4469-88ce-9f9b3f9097ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:53:51.613815 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 764/1512] 127.0.0.1 () {58 vars in 998 bytes} [Sun Oct 9 00:53:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:53:51.765192 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2c7bcf6-342e-4664-9ae5-47d37160c493 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:53:51.767898 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-a2c7bcf6-342e-4664-9ae5-47d37160c493 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:53:51.886584 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2c7bcf6-342e-4664-9ae5-47d37160c493 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:53:51.892035 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b33daa8c-dd82-49b5-bf74-418f15c5aa20 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a5d95969e74157bb2d2a57207e1651', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:53:51.892631 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b33daa8c-dd82-49b5-bf74-418f15c5aa20 service nova] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:53:51.892909 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b33daa8c-dd82-49b5-bf74-418f15c5aa20 service nova] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:53:51.893179 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b33daa8c-dd82-49b5-bf74-418f15c5aa20 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:53:51.901043 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b33daa8c-dd82-49b5-bf74-418f15c5aa20 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:53:51.902601 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b33daa8c-dd82-49b5-bf74-418f15c5aa20 service nova] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:53:51.912028 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 749/1513] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:53:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 147 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:53:52.986888 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992c9d4a-fe18-418b-a91a-13eb0d63df07 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:53:52.987426 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992c9d4a-fe18-418b-a91a-13eb0d63df07 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:53:52.987426 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992c9d4a-fe18-418b-a91a-13eb0d63df07 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:53:52.988796 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 765/1514] 127.0.0.1 () {54 vars in 901 bytes} [Sun Oct 9 00:53:52 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:53:52.994847 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f2b154-5803-47ba-b8ee-3e2303db0b65 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:53:52.995278 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f2b154-5803-47ba-b8ee-3e2303db0b65 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:53:52.995397 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f2b154-5803-47ba-b8ee-3e2303db0b65 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:53:53.014479 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-c9f2b154-5803-47ba-b8ee-3e2303db0b65 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:53:53.015692 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-c9f2b154-5803-47ba-b8ee-3e2303db0b65 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:53:53.023381 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 750/1515] 127.0.0.1 () {58 vars in 998 bytes} [Sun Oct 9 00:53:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:53:53.152095 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef5ba048-3f34-4b77-9977-f53b37cb17b6 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:53:53.152497 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef5ba048-3f34-4b77-9977-f53b37cb17b6 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:53:53.153038 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef5ba048-3f34-4b77-9977-f53b37cb17b6 None None] PATH_INFO: `/v3` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:53:53.154343 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 766/1516] 10.222.0.60 () {54 vars in 925 bytes} [Sun Oct 9 00:53:53 2022] GET /identity/v3 => generated 255 bytes in 3 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:53:53.160763 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-758ca453-b813-4b9b-a772-c39502b185ee None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:53:53.161048 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-758ca453-b813-4b9b-a772-c39502b185ee None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:53:53.161322 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-758ca453-b813-4b9b-a772-c39502b185ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:53:53.176895 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef5ba048-3f34-4b77-9977-f53b37cb17b6 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:53:53.190978 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef5ba048-3f34-4b77-9977-f53b37cb17b6 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:53:53.196603 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30b4a0db-eb3c-4875-a792-a2aa7fab0636 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a5d95969e74157bb2d2a57207e1651', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:53:53.197196 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b4a0db-eb3c-4875-a792-a2aa7fab0636 service nova] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:53:53.197483 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b4a0db-eb3c-4875-a792-a2aa7fab0636 service nova] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:53:53.197759 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30b4a0db-eb3c-4875-a792-a2aa7fab0636 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:53:53.206546 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30b4a0db-eb3c-4875-a792-a2aa7fab0636 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:53:53.208174 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30b4a0db-eb3c-4875-a792-a2aa7fab0636 service nova] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:53:53.217643 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 767/1517] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:53:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 09 00:53:53.221836 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-758ca453-b813-4b9b-a772-c39502b185ee None None] MFA Rules not processed for user `d25e240cb93643ed8122ccac9586936f`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:53:53.223024 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-758ca453-b813-4b9b-a772-c39502b185ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:53:53.278769 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:53:53.298927 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 751/1518] 10.222.0.60 () {58 vars in 1015 bytes} [Sun Oct 9 00:53:53 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:54:02.059848 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e70bf4-f161-419f-980b-5a9d3aec94f0 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:54:02.060312 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e70bf4-f161-419f-980b-5a9d3aec94f0 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:54:02.060416 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e70bf4-f161-419f-980b-5a9d3aec94f0 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:54:02.061783 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 768/1519] 10.222.0.33 () {54 vars in 961 bytes} [Sun Oct 9 00:54:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:54:02.067201 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12c1627e-bd98-4ebd-a7bb-51a3b975ec95 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:54:02.067630 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12c1627e-bd98-4ebd-a7bb-51a3b975ec95 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:54:02.067798 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12c1627e-bd98-4ebd-a7bb-51a3b975ec95 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:54:02.087736 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-12c1627e-bd98-4ebd-a7bb-51a3b975ec95 None None] MFA Rules not processed for user `d25e240cb93643ed8122ccac9586936f`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:54:02.088940 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-12c1627e-bd98-4ebd-a7bb-51a3b975ec95 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:54:02.098047 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 752/1520] 10.222.0.33 () {58 vars in 1058 bytes} [Sun Oct 9 00:54:02 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:54:02.105582 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9e70bf4-f161-419f-980b-5a9d3aec94f0 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:54:02.118788 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9e70bf4-f161-419f-980b-5a9d3aec94f0 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:54:02.123576 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f45e2d7-a5ec-4cc3-91fe-2926981f22ed service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd25e240cb93643ed8122ccac9586936f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:54:02.124194 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f45e2d7-a5ec-4cc3-91fe-2926981f22ed service placement] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:54:02.124466 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f45e2d7-a5ec-4cc3-91fe-2926981f22ed service placement] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:54:02.124734 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f45e2d7-a5ec-4cc3-91fe-2926981f22ed service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:54:02.132731 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f45e2d7-a5ec-4cc3-91fe-2926981f22ed service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:54:02.134317 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f45e2d7-a5ec-4cc3-91fe-2926981f22ed service placement] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:54:02.144054 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 769/1521] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:54:02 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:54:04.661754 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce44e94-41e6-4d28-b2ce-49e931e558d1 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:54:04.662172 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce44e94-41e6-4d28-b2ce-49e931e558d1 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:54:04.662312 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce44e94-41e6-4d28-b2ce-49e931e558d1 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:54:04.663706 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 753/1522] 127.0.0.1 () {54 vars in 901 bytes} [Sun Oct 9 00:54:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:54:04.669900 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c2939b8-7115-4b18-a2ea-3b8db74eb372 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:54:04.670308 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c2939b8-7115-4b18-a2ea-3b8db74eb372 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:54:04.670444 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c2939b8-7115-4b18-a2ea-3b8db74eb372 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:54:04.690130 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-7c2939b8-7115-4b18-a2ea-3b8db74eb372 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:54:04.691303 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-7c2939b8-7115-4b18-a2ea-3b8db74eb372 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:54:04.698814 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 770/1523] 127.0.0.1 () {58 vars in 998 bytes} [Sun Oct 9 00:54:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:54:04.868397 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ce44e94-41e6-4d28-b2ce-49e931e558d1 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:54:04.883181 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ce44e94-41e6-4d28-b2ce-49e931e558d1 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:54:04.888693 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-909d4847-ecca-4157-84a7-dd6fea20fc27 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a5d95969e74157bb2d2a57207e1651', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:54:04.889310 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-909d4847-ecca-4157-84a7-dd6fea20fc27 service nova] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:54:04.889586 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-909d4847-ecca-4157-84a7-dd6fea20fc27 service nova] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:54:04.889856 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-909d4847-ecca-4157-84a7-dd6fea20fc27 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:54:04.898845 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-909d4847-ecca-4157-84a7-dd6fea20fc27 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:54:04.900437 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-909d4847-ecca-4157-84a7-dd6fea20fc27 service nova] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:54:04.909838 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 754/1524] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:54:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 09 00:54:05.970447 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c328b3b1-11c8-4da5-ae2c-1e7f00ce657d None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:54:05.971642 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c328b3b1-11c8-4da5-ae2c-1e7f00ce657d None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:54:05.971642 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c328b3b1-11c8-4da5-ae2c-1e7f00ce657d None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:54:05.972962 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 771/1525] 127.0.0.1 () {54 vars in 901 bytes} [Sun Oct 9 00:54:05 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:54:05.979043 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85fbec99-9de9-445d-b303-fb97a5aaa3b0 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:54:05.979459 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85fbec99-9de9-445d-b303-fb97a5aaa3b0 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:54:05.979584 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85fbec99-9de9-445d-b303-fb97a5aaa3b0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:54:05.999799 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-85fbec99-9de9-445d-b303-fb97a5aaa3b0 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:54:06.000986 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-85fbec99-9de9-445d-b303-fb97a5aaa3b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:54:06.008590 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 755/1526] 127.0.0.1 () {58 vars in 998 bytes} [Sun Oct 9 00:54:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:54:06.158843 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c328b3b1-11c8-4da5-ae2c-1e7f00ce657d None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:54:06.172859 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c328b3b1-11c8-4da5-ae2c-1e7f00ce657d None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:54:06.178272 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02da335b-8225-4237-bb41-30d92711bfcc service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a5d95969e74157bb2d2a57207e1651', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:54:06.178879 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02da335b-8225-4237-bb41-30d92711bfcc service nova] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:54:06.179159 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02da335b-8225-4237-bb41-30d92711bfcc service nova] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:54:06.179432 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02da335b-8225-4237-bb41-30d92711bfcc service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:54:06.188321 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02da335b-8225-4237-bb41-30d92711bfcc service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:54:06.190484 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02da335b-8225-4237-bb41-30d92711bfcc service nova] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:54:06.200934 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 772/1527] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:54:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:54:10.224185 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22cb9149-7ab4-4c66-ad10-ff5d2d5652cd None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:54:10.224713 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22cb9149-7ab4-4c66-ad10-ff5d2d5652cd None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:54:10.224713 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22cb9149-7ab4-4c66-ad10-ff5d2d5652cd None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:54:10.226051 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 756/1528] 127.0.0.1 () {54 vars in 917 bytes} [Sun Oct 9 00:54:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:54:10.231258 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15475a2c-45e7-4ff3-8aca-27b3e3b61363 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:54:10.231707 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15475a2c-45e7-4ff3-8aca-27b3e3b61363 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:54:10.231848 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15475a2c-45e7-4ff3-8aca-27b3e3b61363 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:54:10.251139 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-15475a2c-45e7-4ff3-8aca-27b3e3b61363 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:54:10.252885 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-15475a2c-45e7-4ff3-8aca-27b3e3b61363 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:54:10.260634 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 773/1529] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Oct 9 00:54:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:54:10.555671 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22cb9149-7ab4-4c66-ad10-ff5d2d5652cd None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:54:10.557394 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-22cb9149-7ab4-4c66-ad10-ff5d2d5652cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:54:10.633821 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22cb9149-7ab4-4c66-ad10-ff5d2d5652cd None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:54:10.637735 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-756ccf33-e052-4e8c-a611-493881be2cff service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:54:10.638329 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-756ccf33-e052-4e8c-a611-493881be2cff service neutron] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:54:10.638614 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-756ccf33-e052-4e8c-a611-493881be2cff service neutron] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:54:10.638886 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-756ccf33-e052-4e8c-a611-493881be2cff service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:54:10.647561 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-756ccf33-e052-4e8c-a611-493881be2cff service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:54:10.649010 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-756ccf33-e052-4e8c-a611-493881be2cff service neutron] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:54:10.658945 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 757/1530] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:54:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:54:12.807565 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85cb916-cf01-4111-ac33-18a570e663c4 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:54:12.808121 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85cb916-cf01-4111-ac33-18a570e663c4 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:54:12.808121 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85cb916-cf01-4111-ac33-18a570e663c4 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:54:12.809491 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 774/1531] 127.0.0.1 () {54 vars in 917 bytes} [Sun Oct 9 00:54:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:54:12.816256 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-267e9ab5-d001-420d-8fda-31a6001ae55f None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:54:12.816736 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-267e9ab5-d001-420d-8fda-31a6001ae55f None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:54:12.816962 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-267e9ab5-d001-420d-8fda-31a6001ae55f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:54:12.838459 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-267e9ab5-d001-420d-8fda-31a6001ae55f None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:54:12.839698 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-267e9ab5-d001-420d-8fda-31a6001ae55f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:54:12.848176 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 758/1532] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Oct 9 00:54:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:54:13.132500 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a85cb916-cf01-4111-ac33-18a570e663c4 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:54:13.137792 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=94 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:54:13.145082 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a85cb916-cf01-4111-ac33-18a570e663c4 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:54:13.148575 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e8d2c71-b8da-437d-9e41-ea96858e5a54 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:54:13.149163 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e8d2c71-b8da-437d-9e41-ea96858e5a54 service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:54:13.149442 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e8d2c71-b8da-437d-9e41-ea96858e5a54 service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:54:13.149714 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e8d2c71-b8da-437d-9e41-ea96858e5a54 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:54:13.158730 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e8d2c71-b8da-437d-9e41-ea96858e5a54 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:54:13.160190 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e8d2c71-b8da-437d-9e41-ea96858e5a54 service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:54:13.169312 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 775/1533] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:54:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:54:16.823969 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-200440f1-f9a0-4ea1-b503-2b9c3c48ac88 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:54:16.824557 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-200440f1-f9a0-4ea1-b503-2b9c3c48ac88 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:54:16.824557 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-200440f1-f9a0-4ea1-b503-2b9c3c48ac88 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:54:16.825836 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 759/1534] 127.0.0.1 () {54 vars in 917 bytes} [Sun Oct 9 00:54:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:54:16.831582 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99bfcc35-0ddc-46c4-8f06-ac68aea9040d None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:54:16.832045 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99bfcc35-0ddc-46c4-8f06-ac68aea9040d None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:54:16.832146 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99bfcc35-0ddc-46c4-8f06-ac68aea9040d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:54:16.852481 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-99bfcc35-0ddc-46c4-8f06-ac68aea9040d None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:54:16.853678 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-99bfcc35-0ddc-46c4-8f06-ac68aea9040d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:54:16.861727 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 776/1535] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Oct 9 00:54:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:54:17.164537 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-200440f1-f9a0-4ea1-b503-2b9c3c48ac88 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:54:17.166214 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-200440f1-f9a0-4ea1-b503-2b9c3c48ac88 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:54:17.241731 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-200440f1-f9a0-4ea1-b503-2b9c3c48ac88 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:54:17.245248 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05ddf22d-f812-45a2-aa04-dc06263e9e08 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3f48335668a4e8f934ac11ff66cd308', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:54:17.245839 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05ddf22d-f812-45a2-aa04-dc06263e9e08 service cinder] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:54:17.246119 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05ddf22d-f812-45a2-aa04-dc06263e9e08 service cinder] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:54:17.246397 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05ddf22d-f812-45a2-aa04-dc06263e9e08 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:54:17.255034 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05ddf22d-f812-45a2-aa04-dc06263e9e08 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:54:17.256478 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05ddf22d-f812-45a2-aa04-dc06263e9e08 service cinder] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:54:17.266057 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 760/1536] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:54:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:54:19.394681 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3f7f30e-dfb7-412f-899f-56a3cc02ee6c None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:54:19.395194 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3f7f30e-dfb7-412f-899f-56a3cc02ee6c None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:54:19.395194 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3f7f30e-dfb7-412f-899f-56a3cc02ee6c None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:54:19.396521 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 777/1537] 127.0.0.1 () {54 vars in 917 bytes} [Sun Oct 9 00:54:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:54:19.401633 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-170a1448-afd4-4fde-8869-33b5976d2be2 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:54:19.402027 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-170a1448-afd4-4fde-8869-33b5976d2be2 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:54:19.402181 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-170a1448-afd4-4fde-8869-33b5976d2be2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:54:19.421596 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-170a1448-afd4-4fde-8869-33b5976d2be2 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:54:19.422801 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-170a1448-afd4-4fde-8869-33b5976d2be2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:54:19.431014 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 761/1538] 127.0.0.1 () {58 vars in 1014 bytes} [Sun Oct 9 00:54:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:54:19.729960 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3f7f30e-dfb7-412f-899f-56a3cc02ee6c None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:54:19.731655 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-d3f7f30e-dfb7-412f-899f-56a3cc02ee6c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:54:19.745362 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3f7f30e-dfb7-412f-899f-56a3cc02ee6c None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:54:19.748834 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ed3e53e-3984-42bc-bc75-4f1689579e86 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3f48335668a4e8f934ac11ff66cd308', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:54:19.749421 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed3e53e-3984-42bc-bc75-4f1689579e86 service cinder] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:54:19.749699 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed3e53e-3984-42bc-bc75-4f1689579e86 service cinder] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:54:19.749971 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed3e53e-3984-42bc-bc75-4f1689579e86 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:54:19.758619 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ed3e53e-3984-42bc-bc75-4f1689579e86 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:54:19.760074 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ed3e53e-3984-42bc-bc75-4f1689579e86 service cinder] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:54:19.769168 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 778/1539] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:54:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:54:29.416709 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=92 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:54:29.765188 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=14 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:55:43.692491 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0556975b-d591-4c35-a24d-9d9d59f8db82 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:43.708067 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0556975b-d591-4c35-a24d-9d9d59f8db82 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:43.708067 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0556975b-d591-4c35-a24d-9d9d59f8db82 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:43.715431 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-0556975b-d591-4c35-a24d-9d9d59f8db82 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:55:43.716763 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-0556975b-d591-4c35-a24d-9d9d59f8db82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:55:43.731009 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 762/1540] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:55:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:55:43.740942 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dbd72d2-795d-4901-ad0b-da8fd7cf2e08 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:43.741509 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dbd72d2-795d-4901-ad0b-da8fd7cf2e08 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:43.741509 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dbd72d2-795d-4901-ad0b-da8fd7cf2e08 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:43.769011 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-8dbd72d2-795d-4901-ad0b-da8fd7cf2e08 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:55:43.770517 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-8dbd72d2-795d-4901-ad0b-da8fd7cf2e08 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:55:43.786330 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 779/1541] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:55:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:55:43.791321 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0556975b-d591-4c35-a24d-9d9d59f8db82 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:43.805618 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0556975b-d591-4c35-a24d-9d9d59f8db82 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:43.816729 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da83f77b-41ab-469a-bc34-82c77140cb59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:43.817366 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da83f77b-41ab-469a-bc34-82c77140cb59 admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:43.817657 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da83f77b-41ab-469a-bc34-82c77140cb59 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:43.817949 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da83f77b-41ab-469a-bc34-82c77140cb59 admin admin] PATH_INFO: `/v3/domains` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:43.818603 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da83f77b-41ab-469a-bc34-82c77140cb59 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:55:43.818908 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da83f77b-41ab-469a-bc34-82c77140cb59 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:43.819965 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: WARNING py.warnings [None req-da83f77b-41ab-469a-bc34-82c77140cb59 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 09 00:55:43.819965 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: warnings.warn(msg) Oct 09 00:55:43.819965 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: Oct 09 00:55:43.820877 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da83f77b-41ab-469a-bc34-82c77140cb59 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:43.839677 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 763/1542] 10.222.0.33 () {58 vars in 1153 bytes} [Sun Oct 9 00:55:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:55:43.849603 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dbd72d2-795d-4901-ad0b-da8fd7cf2e08 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:43.863180 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dbd72d2-795d-4901-ad0b-da8fd7cf2e08 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:43.871260 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d6ccd9d-8b33-4e17-9db8-bc1837c62165 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:43.872424 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d6ccd9d-8b33-4e17-9db8-bc1837c62165 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:43.873124 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d6ccd9d-8b33-4e17-9db8-bc1837c62165 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:43.873622 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d6ccd9d-8b33-4e17-9db8-bc1837c62165 admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:43.874660 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d6ccd9d-8b33-4e17-9db8-bc1837c62165 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:43.877494 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: WARNING py.warnings [None req-3d6ccd9d-8b33-4e17-9db8-bc1837c62165 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 09 00:55:43.877494 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: warnings.warn(msg) Oct 09 00:55:43.877494 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: Oct 09 00:55:43.879299 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d6ccd9d-8b33-4e17-9db8-bc1837c62165 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:43.944163 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 780/1543] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 00:55:43 2022] POST /identity/v3/projects => generated 377 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:55:43.944580 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c7154e-fb77-48cd-8e35-ceb22154096f None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:43.944872 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c7154e-fb77-48cd-8e35-ceb22154096f None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:43.945057 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c7154e-fb77-48cd-8e35-ceb22154096f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:43.948630 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d6ccd9d-8b33-4e17-9db8-bc1837c62165 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:43.981932 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-d1c7154e-fb77-48cd-8e35-ceb22154096f None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:55:43.983211 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-d1c7154e-fb77-48cd-8e35-ceb22154096f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:55:44.048523 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d6ccd9d-8b33-4e17-9db8-bc1837c62165 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:44.064438 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cde5a76-c5a6-4359-ad38-c69c609b8ee4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:44.065477 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cde5a76-c5a6-4359-ad38-c69c609b8ee4 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:44.066007 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cde5a76-c5a6-4359-ad38-c69c609b8ee4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:44.066505 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cde5a76-c5a6-4359-ad38-c69c609b8ee4 admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:44.067388 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cde5a76-c5a6-4359-ad38-c69c609b8ee4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:44.069671 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cde5a76-c5a6-4359-ad38-c69c609b8ee4 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:44.127217 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 781/1544] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:55:43 2022] POST /identity/v3/users => generated 289 bytes in 175 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 09 00:55:44.127641 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef721ce3-2111-4594-bcef-af21c8bc8566 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:44.128041 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef721ce3-2111-4594-bcef-af21c8bc8566 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:44.128453 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef721ce3-2111-4594-bcef-af21c8bc8566 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:44.134791 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 764/1545] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:55:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 09 00:55:44.135881 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1c7154e-fb77-48cd-8e35-ceb22154096f None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:44.153145 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1c7154e-fb77-48cd-8e35-ceb22154096f None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:44.163768 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b33cf450-68c1-40c7-baba-d0a89ad250dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:44.164888 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b33cf450-68c1-40c7-baba-d0a89ad250dc admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:44.165338 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b33cf450-68c1-40c7-baba-d0a89ad250dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:44.165777 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b33cf450-68c1-40c7-baba-d0a89ad250dc admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:44.166591 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b33cf450-68c1-40c7-baba-d0a89ad250dc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:44.168074 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-ef721ce3-2111-4594-bcef-af21c8bc8566 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:55:44.168591 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b33cf450-68c1-40c7-baba-d0a89ad250dc admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:44.169528 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-ef721ce3-2111-4594-bcef-af21c8bc8566 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:55:44.184809 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 765/1546] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:55:44 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:55:44.186356 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dfdb282-ad01-4041-88ca-e7f6e0d49a29 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:44.186821 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dfdb282-ad01-4041-88ca-e7f6e0d49a29 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:44.187329 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dfdb282-ad01-4041-88ca-e7f6e0d49a29 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:44.208856 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 782/1547] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:55:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:55:44.209843 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef721ce3-2111-4594-bcef-af21c8bc8566 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:44.222220 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-6dfdb282-ad01-4041-88ca-e7f6e0d49a29 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:55:44.223813 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-6dfdb282-ad01-4041-88ca-e7f6e0d49a29 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:55:44.230217 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef721ce3-2111-4594-bcef-af21c8bc8566 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:44.239066 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36a2e1d2-881a-431d-a6ea-b5a03e965417 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:44.239781 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a2e1d2-881a-431d-a6ea-b5a03e965417 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:44.240067 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a2e1d2-881a-431d-a6ea-b5a03e965417 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:44.240343 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a2e1d2-881a-431d-a6ea-b5a03e965417 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:44.240997 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36a2e1d2-881a-431d-a6ea-b5a03e965417 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:44.242612 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36a2e1d2-881a-431d-a6ea-b5a03e965417 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:44.245512 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 766/1548] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:55:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 60 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:55:44.246779 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8dfce7b-3515-4797-9a51-8e02102cbe60 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:44.247065 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8dfce7b-3515-4797-9a51-8e02102cbe60 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:44.247340 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8dfce7b-3515-4797-9a51-8e02102cbe60 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:44.264127 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 783/1549] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:55:44 2022] GET /identity/v3/roles => generated 1420 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:55:44.266334 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2058fab6-9201-4594-ba72-a05a3e78894f None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:44.266470 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2058fab6-9201-4594-ba72-a05a3e78894f None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:44.266682 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2058fab6-9201-4594-ba72-a05a3e78894f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:44.284605 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-e8dfce7b-3515-4797-9a51-8e02102cbe60 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:55:44.286176 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-e8dfce7b-3515-4797-9a51-8e02102cbe60 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:55:44.300410 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 767/1550] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:55:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 55 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:55:44.302066 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8dfce7b-3515-4797-9a51-8e02102cbe60 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:44.311764 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-2058fab6-9201-4594-ba72-a05a3e78894f None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:55:44.313451 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-2058fab6-9201-4594-ba72-a05a3e78894f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:55:44.321369 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8dfce7b-3515-4797-9a51-8e02102cbe60 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:44.329558 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 784/1551] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:55:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 65 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:55:44.333120 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b52bfaa-8619-45d6-8e42-9b1f97c8e1cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:44.333387 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6719bd-a4ed-4930-9e5f-f5616abcfaf0 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:44.333745 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b52bfaa-8619-45d6-8e42-9b1f97c8e1cf admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:44.334033 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b52bfaa-8619-45d6-8e42-9b1f97c8e1cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:44.334314 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b52bfaa-8619-45d6-8e42-9b1f97c8e1cf admin admin] PATH_INFO: `/v3/domains` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:44.334563 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6719bd-a4ed-4930-9e5f-f5616abcfaf0 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:44.334969 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b52bfaa-8619-45d6-8e42-9b1f97c8e1cf admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:55:44.335264 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6719bd-a4ed-4930-9e5f-f5616abcfaf0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:44.335390 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b52bfaa-8619-45d6-8e42-9b1f97c8e1cf admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:44.336967 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b52bfaa-8619-45d6-8e42-9b1f97c8e1cf admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:44.350537 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 768/1552] 10.222.0.33 () {58 vars in 1153 bytes} [Sun Oct 9 00:55:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:55:44.355711 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b52bfaa-8619-45d6-8e42-9b1f97c8e1cf admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:44.358610 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-fd6719bd-a4ed-4930-9e5f-f5616abcfaf0 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:55:44.359827 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-fd6719bd-a4ed-4930-9e5f-f5616abcfaf0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:55:44.365324 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b52bfaa-8619-45d6-8e42-9b1f97c8e1cf admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:44.369564 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 785/1553] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:55:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:55:44.370212 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd6719bd-a4ed-4930-9e5f-f5616abcfaf0 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:44.374792 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d70a71ab-cfbc-4d1d-b5a7-ab7a01bc5643 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:44.375626 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d70a71ab-cfbc-4d1d-b5a7-ab7a01bc5643 admin admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:44.376030 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d70a71ab-cfbc-4d1d-b5a7-ab7a01bc5643 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:44.376455 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d70a71ab-cfbc-4d1d-b5a7-ab7a01bc5643 admin admin] PATH_INFO: `/v3/projects/40e631c5e320489cac9cea85118c6263/users/1512c91201f14b75a9be35b89c2097ef/roles/588016933bab436d8b7c3ac76827356d` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:44.382411 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd6719bd-a4ed-4930-9e5f-f5616abcfaf0 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:44.387864 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94775b2c-5a27-4235-a7f1-80907f9a1a45 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:44.388463 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94775b2c-5a27-4235-a7f1-80907f9a1a45 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:44.388743 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94775b2c-5a27-4235-a7f1-80907f9a1a45 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:44.389012 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94775b2c-5a27-4235-a7f1-80907f9a1a45 admin admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:44.389826 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94775b2c-5a27-4235-a7f1-80907f9a1a45 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:55:44.390471 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94775b2c-5a27-4235-a7f1-80907f9a1a45 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:44.392154 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94775b2c-5a27-4235-a7f1-80907f9a1a45 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:44.393669 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d70a71ab-cfbc-4d1d-b5a7-ab7a01bc5643 admin admin] RBAC: Authorizing `identity:create_grant(project_id=40e631c5e320489cac9cea85118c6263, user_id=1512c91201f14b75a9be35b89c2097ef, role_id=588016933bab436d8b7c3ac76827356d)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:44.396535 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d70a71ab-cfbc-4d1d-b5a7-ab7a01bc5643 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:44.411808 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 786/1554] 10.222.0.33 () {58 vars in 1153 bytes} [Sun Oct 9 00:55:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:55:44.412527 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 769/1555] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:55:44 2022] PUT /identity/v3/projects/40e631c5e320489cac9cea85118c6263/users/1512c91201f14b75a9be35b89c2097ef/roles/588016933bab436d8b7c3ac76827356d => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:55:44.413573 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15d7af0c-96fb-401b-9632-289de7de9db2 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:44.413959 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15d7af0c-96fb-401b-9632-289de7de9db2 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:44.414415 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15d7af0c-96fb-401b-9632-289de7de9db2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:44.414688 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3049f21f-a264-476e-b838-356058f755d1 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:44.415244 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3049f21f-a264-476e-b838-356058f755d1 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:44.415519 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3049f21f-a264-476e-b838-356058f755d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:44.434659 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-15d7af0c-96fb-401b-9632-289de7de9db2 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:55:44.435182 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-3049f21f-a264-476e-b838-356058f755d1 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:55:44.436028 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-15d7af0c-96fb-401b-9632-289de7de9db2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:55:44.436389 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-3049f21f-a264-476e-b838-356058f755d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:55:44.494906 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 787/1556] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:55:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:55:44.495489 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 770/1557] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:55:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:55:44.496092 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c5bdd9b-282a-458c-815c-9e38d63f454d None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:44.496377 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3049f21f-a264-476e-b838-356058f755d1 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:44.496502 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c5bdd9b-282a-458c-815c-9e38d63f454d None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:44.496649 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c5bdd9b-282a-458c-815c-9e38d63f454d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:44.507364 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3049f21f-a264-476e-b838-356058f755d1 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:44.512137 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f16c63e0-cc47-4f44-9ff6-093aec597fdf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:44.512744 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f16c63e0-cc47-4f44-9ff6-093aec597fdf admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:44.513028 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f16c63e0-cc47-4f44-9ff6-093aec597fdf admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:44.513302 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f16c63e0-cc47-4f44-9ff6-093aec597fdf admin admin] PATH_INFO: `/v3/projects` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:44.513929 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f16c63e0-cc47-4f44-9ff6-093aec597fdf admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:44.515959 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f16c63e0-cc47-4f44-9ff6-093aec597fdf admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:44.524051 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-6c5bdd9b-282a-458c-815c-9e38d63f454d None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:55:44.525240 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-6c5bdd9b-282a-458c-815c-9e38d63f454d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:55:44.533405 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 788/1558] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:55:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:55:44.537083 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ea0589-5f28-4c92-a1df-0d821ef37cbe None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:44.537489 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ea0589-5f28-4c92-a1df-0d821ef37cbe None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:44.537939 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ea0589-5f28-4c92-a1df-0d821ef37cbe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:44.539265 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:55:44.546699 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:55:44.548087 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 771/1559] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 00:55:44 2022] POST /identity/v3/projects => generated 365 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:55:44.548843 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f16c63e0-cc47-4f44-9ff6-093aec597fdf admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:44.561105 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-e8ea0589-5f28-4c92-a1df-0d821ef37cbe None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:55:44.562368 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-e8ea0589-5f28-4c92-a1df-0d821ef37cbe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:55:44.610512 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f16c63e0-cc47-4f44-9ff6-093aec597fdf admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:44.615254 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f77aa23e-40d6-4521-82f4-a00b7c8b00a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:44.615881 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f77aa23e-40d6-4521-82f4-a00b7c8b00a5 admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:44.616226 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f77aa23e-40d6-4521-82f4-a00b7c8b00a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:44.616448 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f77aa23e-40d6-4521-82f4-a00b7c8b00a5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:44.617073 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f77aa23e-40d6-4521-82f4-a00b7c8b00a5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:44.618973 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f77aa23e-40d6-4521-82f4-a00b7c8b00a5 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:44.622693 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 789/1560] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:55:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:55:44.623958 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7367f51-7c23-4644-9400-5c1f9fa6f549 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:44.624251 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7367f51-7c23-4644-9400-5c1f9fa6f549 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:44.624521 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7367f51-7c23-4644-9400-5c1f9fa6f549 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:44.644703 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-c7367f51-7c23-4644-9400-5c1f9fa6f549 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:55:44.645891 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-c7367f51-7c23-4644-9400-5c1f9fa6f549 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:55:44.650097 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 772/1561] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 00:55:44 2022] POST /identity/v3/projects => generated 377 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:55:44.650097 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97591020-fad7-405f-aa7d-1fb5bfc19f09 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:44.650097 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97591020-fad7-405f-aa7d-1fb5bfc19f09 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:44.650097 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97591020-fad7-405f-aa7d-1fb5bfc19f09 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:44.667466 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-97591020-fad7-405f-aa7d-1fb5bfc19f09 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:55:44.668697 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-97591020-fad7-405f-aa7d-1fb5bfc19f09 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:55:44.708233 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 790/1562] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:55:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:55:44.709703 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3d3c0ee-017f-4066-b7fa-be428324a78e None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:44.710094 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3d3c0ee-017f-4066-b7fa-be428324a78e None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:44.710477 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3d3c0ee-017f-4066-b7fa-be428324a78e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:44.730779 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 773/1563] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:55:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:55:44.732131 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd54eec8-bf9e-4041-89c0-6116be97b637 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:44.732417 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd54eec8-bf9e-4041-89c0-6116be97b637 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:44.732616 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-c3d3c0ee-017f-4066-b7fa-be428324a78e None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:55:44.732910 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd54eec8-bf9e-4041-89c0-6116be97b637 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:44.733851 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-c3d3c0ee-017f-4066-b7fa-be428324a78e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:55:44.742693 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 791/1564] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:55:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:55:44.742693 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3d3c0ee-017f-4066-b7fa-be428324a78e None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:44.754032 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3d3c0ee-017f-4066-b7fa-be428324a78e None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:44.754799 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-cd54eec8-bf9e-4041-89c0-6116be97b637 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:55:44.756060 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-cd54eec8-bf9e-4041-89c0-6116be97b637 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:55:44.758603 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38b8c671-512b-4728-8c8e-ab01838b2bc9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:44.759179 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b8c671-512b-4728-8c8e-ab01838b2bc9 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:44.759462 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b8c671-512b-4728-8c8e-ab01838b2bc9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:44.759771 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b8c671-512b-4728-8c8e-ab01838b2bc9 admin admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:44.760404 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38b8c671-512b-4728-8c8e-ab01838b2bc9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:55:44.760708 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38b8c671-512b-4728-8c8e-ab01838b2bc9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:44.762315 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38b8c671-512b-4728-8c8e-ab01838b2bc9 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:44.764083 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 774/1565] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:55:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:55:44.768399 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd54eec8-bf9e-4041-89c0-6116be97b637 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:44.784270 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 792/1566] 10.222.0.33 () {58 vars in 1153 bytes} [Sun Oct 9 00:55:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:55:44.784454 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd54eec8-bf9e-4041-89c0-6116be97b637 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:44.784553 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38b8c671-512b-4728-8c8e-ab01838b2bc9 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:44.788271 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38b8c671-512b-4728-8c8e-ab01838b2bc9 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:44.789044 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66f085df-3bc9-45b9-8ad6-ed48e784e071 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:44.789794 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f085df-3bc9-45b9-8ad6-ed48e784e071 admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:44.790076 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f085df-3bc9-45b9-8ad6-ed48e784e071 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:44.790344 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f085df-3bc9-45b9-8ad6-ed48e784e071 admin admin] PATH_INFO: `/v3/users` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:44.790971 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66f085df-3bc9-45b9-8ad6-ed48e784e071 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:44.792008 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: WARNING py.warnings [None req-66f085df-3bc9-45b9-8ad6-ed48e784e071 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 09 00:55:44.792008 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: warnings.warn(msg) Oct 09 00:55:44.792008 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: Oct 09 00:55:44.792795 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0638e036-b876-4ca6-95cf-decf39aff08d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:44.793212 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66f085df-3bc9-45b9-8ad6-ed48e784e071 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:44.793371 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0638e036-b876-4ca6-95cf-decf39aff08d admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:44.793640 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0638e036-b876-4ca6-95cf-decf39aff08d admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:44.793909 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0638e036-b876-4ca6-95cf-decf39aff08d admin admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:44.794530 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0638e036-b876-4ca6-95cf-decf39aff08d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:55:44.794829 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0638e036-b876-4ca6-95cf-decf39aff08d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:44.796454 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0638e036-b876-4ca6-95cf-decf39aff08d admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:44.810411 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 793/1567] 10.222.0.33 () {58 vars in 1153 bytes} [Sun Oct 9 00:55:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:55:44.811524 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0638e036-b876-4ca6-95cf-decf39aff08d admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:44.819132 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 775/1568] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:55:44 2022] POST /identity/v3/users => generated 283 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:55:44.819132 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb800898-a33a-4461-8ceb-271fd8feebfe None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:44.819132 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb800898-a33a-4461-8ceb-271fd8feebfe None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:44.819132 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb800898-a33a-4461-8ceb-271fd8feebfe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:44.825581 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0638e036-b876-4ca6-95cf-decf39aff08d admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:44.830223 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58cb6aa3-a68b-4e2b-a296-069b6f5783d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:44.830821 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58cb6aa3-a68b-4e2b-a296-069b6f5783d4 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:44.831102 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58cb6aa3-a68b-4e2b-a296-069b6f5783d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:44.831374 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58cb6aa3-a68b-4e2b-a296-069b6f5783d4 admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:44.832038 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58cb6aa3-a68b-4e2b-a296-069b6f5783d4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:44.833951 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58cb6aa3-a68b-4e2b-a296-069b6f5783d4 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:44.839686 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-fb800898-a33a-4461-8ceb-271fd8feebfe None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:55:44.840926 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-fb800898-a33a-4461-8ceb-271fd8feebfe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:55:44.849340 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 776/1569] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:55:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:55:44.850140 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb800898-a33a-4461-8ceb-271fd8feebfe None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:44.857486 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 794/1570] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:55:44 2022] POST /identity/v3/users => generated 289 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:55:44.858349 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58cb6aa3-a68b-4e2b-a296-069b6f5783d4 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:44.868332 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58cb6aa3-a68b-4e2b-a296-069b6f5783d4 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:44.872843 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eb8ba45-3d0e-4950-9801-a2f4b9170d65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:44.873418 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eb8ba45-3d0e-4950-9801-a2f4b9170d65 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:44.873692 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eb8ba45-3d0e-4950-9801-a2f4b9170d65 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:44.873962 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eb8ba45-3d0e-4950-9801-a2f4b9170d65 admin admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:44.874612 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eb8ba45-3d0e-4950-9801-a2f4b9170d65 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:55:44.874909 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eb8ba45-3d0e-4950-9801-a2f4b9170d65 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:44.877137 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eb8ba45-3d0e-4950-9801-a2f4b9170d65 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:44.889029 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 795/1571] 10.222.0.33 () {58 vars in 1153 bytes} [Sun Oct 9 00:55:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:55:44.889830 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eb8ba45-3d0e-4950-9801-a2f4b9170d65 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:44.899685 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eb8ba45-3d0e-4950-9801-a2f4b9170d65 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:44.904581 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34133362-3805-4ef3-a274-4111a6e49d81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:44.905165 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34133362-3805-4ef3-a274-4111a6e49d81 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:44.905446 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34133362-3805-4ef3-a274-4111a6e49d81 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:44.905723 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34133362-3805-4ef3-a274-4111a6e49d81 admin admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:44.906013 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb800898-a33a-4461-8ceb-271fd8feebfe None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:44.906353 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34133362-3805-4ef3-a274-4111a6e49d81 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:55:44.906652 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34133362-3805-4ef3-a274-4111a6e49d81 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:44.908307 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34133362-3805-4ef3-a274-4111a6e49d81 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:44.909669 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b823fc43-f0ba-4db8-825d-155fe2b09822 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:44.910271 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b823fc43-f0ba-4db8-825d-155fe2b09822 service neutron] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:44.910732 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b823fc43-f0ba-4db8-825d-155fe2b09822 service neutron] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:44.911012 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b823fc43-f0ba-4db8-825d-155fe2b09822 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:44.918408 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:55:44.927876 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 796/1572] 10.222.0.33 () {58 vars in 1153 bytes} [Sun Oct 9 00:55:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:55:44.928090 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b823fc43-f0ba-4db8-825d-155fe2b09822 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:44.928195 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34133362-3805-4ef3-a274-4111a6e49d81 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:44.928297 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b823fc43-f0ba-4db8-825d-155fe2b09822 service neutron] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:44.931786 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34133362-3805-4ef3-a274-4111a6e49d81 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:44.935921 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 777/1573] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:55:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:55:44.936474 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-620859db-5538-4132-8001-3cd88dfe135b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:44.936767 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b823fc43-f0ba-4db8-825d-155fe2b09822 service neutron] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:44.937061 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-620859db-5538-4132-8001-3cd88dfe135b admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:44.937338 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-620859db-5538-4132-8001-3cd88dfe135b admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:44.937606 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-620859db-5538-4132-8001-3cd88dfe135b admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:44.938226 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-620859db-5538-4132-8001-3cd88dfe135b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:44.940715 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-620859db-5538-4132-8001-3cd88dfe135b admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:44.946519 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b823fc43-f0ba-4db8-825d-155fe2b09822 service neutron] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:44.951511 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9f475a9-4ae9-48ec-9e97-922ad5025d03 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:44.952127 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9f475a9-4ae9-48ec-9e97-922ad5025d03 admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:44.952398 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9f475a9-4ae9-48ec-9e97-922ad5025d03 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:44.952666 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9f475a9-4ae9-48ec-9e97-922ad5025d03 admin admin] PATH_INFO: `/v3/projects` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:44.953280 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9f475a9-4ae9-48ec-9e97-922ad5025d03 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:44.955204 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9f475a9-4ae9-48ec-9e97-922ad5025d03 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:44.976177 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 797/1574] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 00:55:44 2022] POST /identity/v3/projects => generated 373 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:55:44.977087 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-620859db-5538-4132-8001-3cd88dfe135b admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:44.983658 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:55:44.991638 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 778/1575] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 00:55:44 2022] POST /identity/v3/projects => generated 385 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:55:44.992256 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9f475a9-4ae9-48ec-9e97-922ad5025d03 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:45.056235 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-620859db-5538-4132-8001-3cd88dfe135b admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:45.072149 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9f475a9-4ae9-48ec-9e97-922ad5025d03 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:45.079148 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c874520-8ce6-4172-a737-a48569fd6932 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:45.083898 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c874520-8ce6-4172-a737-a48569fd6932 admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:45.084293 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c874520-8ce6-4172-a737-a48569fd6932 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:45.084642 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c874520-8ce6-4172-a737-a48569fd6932 admin admin] PATH_INFO: `/v3/domains` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:45.085608 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c874520-8ce6-4172-a737-a48569fd6932 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:55:45.086138 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c874520-8ce6-4172-a737-a48569fd6932 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:45.087854 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c874520-8ce6-4172-a737-a48569fd6932 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:45.100834 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 779/1576] 10.222.0.33 () {58 vars in 1153 bytes} [Sun Oct 9 00:55:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 109 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:55:45.104829 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c874520-8ce6-4172-a737-a48569fd6932 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:45.122154 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c874520-8ce6-4172-a737-a48569fd6932 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:45.124930 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-757bc5e5-6831-4ad0-a468-f2f40e607646 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:45.125610 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-757bc5e5-6831-4ad0-a468-f2f40e607646 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:45.126046 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-757bc5e5-6831-4ad0-a468-f2f40e607646 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:45.129601 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-757bc5e5-6831-4ad0-a468-f2f40e607646 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:45.129601 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-757bc5e5-6831-4ad0-a468-f2f40e607646 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:45.129601 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-757bc5e5-6831-4ad0-a468-f2f40e607646 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:45.133855 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5844406-f618-4a87-8b56-f21032242d37 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:45.135518 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5844406-f618-4a87-8b56-f21032242d37 admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:45.136022 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5844406-f618-4a87-8b56-f21032242d37 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:45.136702 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5844406-f618-4a87-8b56-f21032242d37 admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:45.137845 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5844406-f618-4a87-8b56-f21032242d37 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:45.140513 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5844406-f618-4a87-8b56-f21032242d37 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:45.143929 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 798/1577] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:55:44 2022] GET /identity/v3/roles => generated 1420 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:55:45.144976 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-757bc5e5-6831-4ad0-a468-f2f40e607646 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:45.157240 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-757bc5e5-6831-4ad0-a468-f2f40e607646 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:45.158677 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 780/1578] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:55:45 2022] GET /identity/v3/roles => generated 1420 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:55:45.159788 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5844406-f618-4a87-8b56-f21032242d37 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:45.164256 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b55b3d1e-a725-4a70-ad72-19b41b135bcd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:45.165126 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b55b3d1e-a725-4a70-ad72-19b41b135bcd admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:45.165515 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b55b3d1e-a725-4a70-ad72-19b41b135bcd admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:45.165862 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b55b3d1e-a725-4a70-ad72-19b41b135bcd admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:45.166563 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b55b3d1e-a725-4a70-ad72-19b41b135bcd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:45.169856 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b55b3d1e-a725-4a70-ad72-19b41b135bcd admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:45.172326 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5844406-f618-4a87-8b56-f21032242d37 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:45.178084 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54bc1080-2155-4708-a7d4-8f1880646ef7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:45.178733 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54bc1080-2155-4708-a7d4-8f1880646ef7 admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:45.179015 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54bc1080-2155-4708-a7d4-8f1880646ef7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:45.179284 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54bc1080-2155-4708-a7d4-8f1880646ef7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:45.179973 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54bc1080-2155-4708-a7d4-8f1880646ef7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:45.182305 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54bc1080-2155-4708-a7d4-8f1880646ef7 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:45.204688 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 799/1579] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 00:55:45 2022] POST /identity/v3/projects => generated 385 bytes in 60 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:55:45.206075 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b55b3d1e-a725-4a70-ad72-19b41b135bcd admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:45.217692 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 781/1580] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 00:55:45 2022] POST /identity/v3/projects => generated 393 bytes in 58 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:55:45.222035 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54bc1080-2155-4708-a7d4-8f1880646ef7 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:45.286765 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b55b3d1e-a725-4a70-ad72-19b41b135bcd admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:45.299763 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54bc1080-2155-4708-a7d4-8f1880646ef7 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:45.305125 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fde3e58-1104-47d0-a504-1231225c5899 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:45.306133 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fde3e58-1104-47d0-a504-1231225c5899 admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:45.306720 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fde3e58-1104-47d0-a504-1231225c5899 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:45.307238 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fde3e58-1104-47d0-a504-1231225c5899 admin admin] PATH_INFO: `/v3/users` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:45.308047 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fde3e58-1104-47d0-a504-1231225c5899 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:45.310035 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fde3e58-1104-47d0-a504-1231225c5899 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:45.333483 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 782/1581] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:55:45 2022] POST /identity/v3/users => generated 293 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:55:45.334663 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fde3e58-1104-47d0-a504-1231225c5899 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:45.339329 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:55:45.349888 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fde3e58-1104-47d0-a504-1231225c5899 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:45.355203 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9dbae6f-7632-4cc3-a595-2bb246822172 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:45.356190 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9dbae6f-7632-4cc3-a595-2bb246822172 admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:45.356637 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9dbae6f-7632-4cc3-a595-2bb246822172 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:45.357070 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9dbae6f-7632-4cc3-a595-2bb246822172 admin admin] PATH_INFO: `/v3/projects` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:45.357697 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9dbae6f-7632-4cc3-a595-2bb246822172 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:45.359840 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9dbae6f-7632-4cc3-a595-2bb246822172 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:45.375833 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-160b0c02-e600-4bcb-ba11-dd21194c48e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:45.376760 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-160b0c02-e600-4bcb-ba11-dd21194c48e7 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:45.377232 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-160b0c02-e600-4bcb-ba11-dd21194c48e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:45.377568 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-160b0c02-e600-4bcb-ba11-dd21194c48e7 admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:45.378284 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-160b0c02-e600-4bcb-ba11-dd21194c48e7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:45.380840 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-160b0c02-e600-4bcb-ba11-dd21194c48e7 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:45.390788 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 783/1582] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 00:55:45 2022] POST /identity/v3/projects => generated 367 bytes in 57 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:55:45.391729 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9dbae6f-7632-4cc3-a595-2bb246822172 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:45.396814 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:55:45.407279 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 800/1583] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:55:45 2022] POST /identity/v3/users => generated 287 bytes in 202 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:55:45.409513 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-160b0c02-e600-4bcb-ba11-dd21194c48e7 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:45.421656 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-160b0c02-e600-4bcb-ba11-dd21194c48e7 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:45.472438 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9dbae6f-7632-4cc3-a595-2bb246822172 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:45.477824 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc2927e7-fab8-4b6c-961d-980e78ffb411 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:45.478463 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc2927e7-fab8-4b6c-961d-980e78ffb411 admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:45.478955 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc2927e7-fab8-4b6c-961d-980e78ffb411 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:45.479240 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc2927e7-fab8-4b6c-961d-980e78ffb411 admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:45.480305 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc2927e7-fab8-4b6c-961d-980e78ffb411 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:45.481895 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc2927e7-fab8-4b6c-961d-980e78ffb411 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:45.487522 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b43f17b5-ff47-43c3-91d1-665584e871f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:45.488926 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b43f17b5-ff47-43c3-91d1-665584e871f8 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:45.489463 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b43f17b5-ff47-43c3-91d1-665584e871f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:45.490507 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b43f17b5-ff47-43c3-91d1-665584e871f8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:45.491476 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b43f17b5-ff47-43c3-91d1-665584e871f8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:45.493642 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b43f17b5-ff47-43c3-91d1-665584e871f8 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:45.493842 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 784/1584] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:55:45 2022] GET /identity/v3/roles => generated 1420 bytes in 102 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:55:45.494713 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc2927e7-fab8-4b6c-961d-980e78ffb411 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:45.509230 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc2927e7-fab8-4b6c-961d-980e78ffb411 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:45.513650 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 801/1585] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:55:45 2022] GET /identity/v3/roles => generated 1420 bytes in 104 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:55:45.513650 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b43f17b5-ff47-43c3-91d1-665584e871f8 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:45.521670 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2828d740-db3d-41ae-a5ab-f8b7b4d47664 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:45.522378 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2828d740-db3d-41ae-a5ab-f8b7b4d47664 admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:45.522902 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2828d740-db3d-41ae-a5ab-f8b7b4d47664 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:45.523164 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2828d740-db3d-41ae-a5ab-f8b7b4d47664 admin admin] PATH_INFO: `/v3/users` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:45.523838 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2828d740-db3d-41ae-a5ab-f8b7b4d47664 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:45.525402 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b43f17b5-ff47-43c3-91d1-665584e871f8 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:45.526133 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2828d740-db3d-41ae-a5ab-f8b7b4d47664 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:45.530640 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e54b8a1-c7bf-4807-9ffa-8c222eac6f9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:45.531442 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e54b8a1-c7bf-4807-9ffa-8c222eac6f9d admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:45.531826 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e54b8a1-c7bf-4807-9ffa-8c222eac6f9d admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:45.532164 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e54b8a1-c7bf-4807-9ffa-8c222eac6f9d admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:45.533532 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e54b8a1-c7bf-4807-9ffa-8c222eac6f9d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:45.535531 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e54b8a1-c7bf-4807-9ffa-8c222eac6f9d admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:45.553759 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 785/1586] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:55:45 2022] POST /identity/v3/users => generated 293 bytes in 59 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:55:45.557523 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2828d740-db3d-41ae-a5ab-f8b7b4d47664 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:45.563060 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 802/1587] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:55:45 2022] POST /identity/v3/users => generated 297 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:55:45.563060 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e54b8a1-c7bf-4807-9ffa-8c222eac6f9d admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:45.567310 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2828d740-db3d-41ae-a5ab-f8b7b4d47664 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:45.572577 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1006ed1b-aece-4269-b964-9c1788b22c46 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:45.573182 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1006ed1b-aece-4269-b964-9c1788b22c46 admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:45.573466 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1006ed1b-aece-4269-b964-9c1788b22c46 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:45.573742 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1006ed1b-aece-4269-b964-9c1788b22c46 admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:45.574377 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1006ed1b-aece-4269-b964-9c1788b22c46 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:45.574491 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e54b8a1-c7bf-4807-9ffa-8c222eac6f9d admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:45.576026 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1006ed1b-aece-4269-b964-9c1788b22c46 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:45.581542 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80e81079-0fe7-47bb-aa25-794d56570564 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:45.582938 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e81079-0fe7-47bb-aa25-794d56570564 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:45.583385 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e81079-0fe7-47bb-aa25-794d56570564 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:45.583860 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e81079-0fe7-47bb-aa25-794d56570564 admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:45.585315 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80e81079-0fe7-47bb-aa25-794d56570564 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:45.587480 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80e81079-0fe7-47bb-aa25-794d56570564 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:45.587909 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 786/1588] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:55:45 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:55:45.588759 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1006ed1b-aece-4269-b964-9c1788b22c46 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:45.603767 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1006ed1b-aece-4269-b964-9c1788b22c46 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:45.609939 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b495b1a-9ffc-4a33-b653-7176a5f1f3e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:45.610727 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b495b1a-9ffc-4a33-b653-7176a5f1f3e2 admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:45.611015 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b495b1a-9ffc-4a33-b653-7176a5f1f3e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:45.611286 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b495b1a-9ffc-4a33-b653-7176a5f1f3e2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:45.612462 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b495b1a-9ffc-4a33-b653-7176a5f1f3e2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:45.614820 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b495b1a-9ffc-4a33-b653-7176a5f1f3e2 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:45.620698 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 803/1589] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:55:45 2022] POST /identity/v3/users => generated 284 bytes in 57 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:55:45.620698 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80e81079-0fe7-47bb-aa25-794d56570564 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:45.625622 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 787/1590] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:55:45 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:55:45.626812 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b495b1a-9ffc-4a33-b653-7176a5f1f3e2 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:45.631986 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80e81079-0fe7-47bb-aa25-794d56570564 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:45.637218 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-462d161a-b55a-448d-9f5a-93dd2d0f7d62 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:45.637935 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-462d161a-b55a-448d-9f5a-93dd2d0f7d62 admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:45.638217 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-462d161a-b55a-448d-9f5a-93dd2d0f7d62 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:45.638505 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-462d161a-b55a-448d-9f5a-93dd2d0f7d62 admin admin] PATH_INFO: `/v3/projects/0ef954aff8eb4042b4fedfc47c6d073e/users/f25b1b172aff4376b6c6905cb3b12913/roles/588016933bab436d8b7c3ac76827356d` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:45.639721 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b495b1a-9ffc-4a33-b653-7176a5f1f3e2 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:45.645642 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-220e2d21-39c9-431a-ab3a-d8d1a8fcbe2a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:45.646368 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-220e2d21-39c9-431a-ab3a-d8d1a8fcbe2a admin admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:45.646639 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-220e2d21-39c9-431a-ab3a-d8d1a8fcbe2a admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:45.647012 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-220e2d21-39c9-431a-ab3a-d8d1a8fcbe2a admin admin] PATH_INFO: `/v3/projects/88e7a07d04d74e739b7fa17f73018b64/users/15e90d2d2b184f339d1d2aacb77bc82b/roles/588016933bab436d8b7c3ac76827356d` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:45.657359 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-462d161a-b55a-448d-9f5a-93dd2d0f7d62 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0ef954aff8eb4042b4fedfc47c6d073e, user_id=f25b1b172aff4376b6c6905cb3b12913, role_id=588016933bab436d8b7c3ac76827356d)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:45.659238 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: WARNING py.warnings [None req-462d161a-b55a-448d-9f5a-93dd2d0f7d62 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 09 00:55:45.659238 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: warnings.warn(msg) Oct 09 00:55:45.659238 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: Oct 09 00:55:45.661442 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-462d161a-b55a-448d-9f5a-93dd2d0f7d62 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:45.675294 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-220e2d21-39c9-431a-ab3a-d8d1a8fcbe2a admin admin] RBAC: Authorizing `identity:create_grant(project_id=88e7a07d04d74e739b7fa17f73018b64, user_id=15e90d2d2b184f339d1d2aacb77bc82b, role_id=588016933bab436d8b7c3ac76827356d)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:45.678052 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-220e2d21-39c9-431a-ab3a-d8d1a8fcbe2a admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:45.679219 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 804/1591] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:55:45 2022] PUT /identity/v3/projects/0ef954aff8eb4042b4fedfc47c6d073e/users/f25b1b172aff4376b6c6905cb3b12913/roles/588016933bab436d8b7c3ac76827356d => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:55:45.680472 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-462d161a-b55a-448d-9f5a-93dd2d0f7d62 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:45.692798 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 788/1592] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:55:45 2022] PUT /identity/v3/projects/88e7a07d04d74e739b7fa17f73018b64/users/15e90d2d2b184f339d1d2aacb77bc82b/roles/588016933bab436d8b7c3ac76827356d => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:55:45.693980 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-220e2d21-39c9-431a-ab3a-d8d1a8fcbe2a admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:45.772764 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-462d161a-b55a-448d-9f5a-93dd2d0f7d62 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:45.777606 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c3b5695-a8c0-4582-b9fa-571ac9db1d2d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:45.778133 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-220e2d21-39c9-431a-ab3a-d8d1a8fcbe2a admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:45.778553 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c3b5695-a8c0-4582-b9fa-571ac9db1d2d admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:45.778553 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c3b5695-a8c0-4582-b9fa-571ac9db1d2d admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:45.778759 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c3b5695-a8c0-4582-b9fa-571ac9db1d2d admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:45.780410 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c3b5695-a8c0-4582-b9fa-571ac9db1d2d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:45.782366 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c3b5695-a8c0-4582-b9fa-571ac9db1d2d admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:45.784701 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=90,keystone:INSERT=10 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:55:45.794509 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb28536e-f08b-4768-b62b-9c42b630d40d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:45.797843 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb28536e-f08b-4768-b62b-9c42b630d40d admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:45.798207 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 805/1593] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:55:45 2022] GET /identity/v3/roles => generated 1420 bytes in 115 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:55:45.798292 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb28536e-f08b-4768-b62b-9c42b630d40d admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:45.798292 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb28536e-f08b-4768-b62b-9c42b630d40d admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:45.798486 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c3b5695-a8c0-4582-b9fa-571ac9db1d2d admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:45.798563 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb28536e-f08b-4768-b62b-9c42b630d40d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:45.799043 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb28536e-f08b-4768-b62b-9c42b630d40d admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:45.806635 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c3b5695-a8c0-4582-b9fa-571ac9db1d2d admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:45.810708 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 789/1594] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:55:45 2022] GET /identity/v3/roles => generated 1420 bytes in 117 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:55:45.811500 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb28536e-f08b-4768-b62b-9c42b630d40d admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:45.815015 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aa5f9da-78d0-4b45-858e-22bc1258342d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:45.815664 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa5f9da-78d0-4b45-858e-22bc1258342d admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:45.815953 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa5f9da-78d0-4b45-858e-22bc1258342d admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:45.816229 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa5f9da-78d0-4b45-858e-22bc1258342d admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:45.816866 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aa5f9da-78d0-4b45-858e-22bc1258342d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:45.818538 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aa5f9da-78d0-4b45-858e-22bc1258342d admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:45.825452 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb28536e-f08b-4768-b62b-9c42b630d40d admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:45.831315 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 806/1595] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:55:45 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:55:45.832540 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aa5f9da-78d0-4b45-858e-22bc1258342d admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:45.832675 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-794ea60a-232a-46b4-8990-e45f4e1e6cc8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:45.832928 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-794ea60a-232a-46b4-8990-e45f4e1e6cc8 admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:45.833209 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-794ea60a-232a-46b4-8990-e45f4e1e6cc8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:45.833479 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-794ea60a-232a-46b4-8990-e45f4e1e6cc8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:45.834501 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-794ea60a-232a-46b4-8990-e45f4e1e6cc8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:45.836718 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-794ea60a-232a-46b4-8990-e45f4e1e6cc8 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:45.843026 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aa5f9da-78d0-4b45-858e-22bc1258342d admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:45.848564 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fe78c48-37fb-45b0-8f1e-6bac6d30180f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:45.849168 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe78c48-37fb-45b0-8f1e-6bac6d30180f admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:45.849451 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe78c48-37fb-45b0-8f1e-6bac6d30180f admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:45.852540 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe78c48-37fb-45b0-8f1e-6bac6d30180f admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:45.852649 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 790/1596] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:55:45 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:55:45.852765 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fe78c48-37fb-45b0-8f1e-6bac6d30180f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:45.852870 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-141a80f7-6052-451f-86c2-c9ba0d5654ee None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:45.852870 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-141a80f7-6052-451f-86c2-c9ba0d5654ee None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:45.852870 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-141a80f7-6052-451f-86c2-c9ba0d5654ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:45.853109 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fe78c48-37fb-45b0-8f1e-6bac6d30180f admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:45.869153 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 807/1597] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:55:45 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:55:45.870489 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a703a90d-cc70-4ec9-ac38-af0b7b493346 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:45.870916 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a703a90d-cc70-4ec9-ac38-af0b7b493346 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:45.871181 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a703a90d-cc70-4ec9-ac38-af0b7b493346 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:45.883078 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-141a80f7-6052-451f-86c2-c9ba0d5654ee None None] MFA Rules not processed for user `f25b1b172aff4376b6c6905cb3b12913`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:55:45.884781 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-141a80f7-6052-451f-86c2-c9ba0d5654ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:55:45.892277 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-a703a90d-cc70-4ec9-ac38-af0b7b493346 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:55:45.893917 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-a703a90d-cc70-4ec9-ac38-af0b7b493346 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:55:45.896634 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:55:45.909874 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 808/1598] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:55:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:55:45.910770 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a703a90d-cc70-4ec9-ac38-af0b7b493346 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:45.922544 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a703a90d-cc70-4ec9-ac38-af0b7b493346 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:45.928890 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-139e0814-4475-4a19-b7cf-bdef33406c2c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:45.929513 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-139e0814-4475-4a19-b7cf-bdef33406c2c admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:45.929787 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-139e0814-4475-4a19-b7cf-bdef33406c2c admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:45.930496 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-139e0814-4475-4a19-b7cf-bdef33406c2c admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:45.930695 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-139e0814-4475-4a19-b7cf-bdef33406c2c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:45.932508 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-139e0814-4475-4a19-b7cf-bdef33406c2c admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:45.948956 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 809/1599] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:55:45 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:55:45.953157 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-139e0814-4475-4a19-b7cf-bdef33406c2c admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:45.967532 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-139e0814-4475-4a19-b7cf-bdef33406c2c admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:45.973545 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e8111de-471f-485c-bc4a-a1a0f4a27251 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:45.974259 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e8111de-471f-485c-bc4a-a1a0f4a27251 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:45.974538 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e8111de-471f-485c-bc4a-a1a0f4a27251 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:45.975185 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e8111de-471f-485c-bc4a-a1a0f4a27251 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:45.975434 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e8111de-471f-485c-bc4a-a1a0f4a27251 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:45.977348 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e8111de-471f-485c-bc4a-a1a0f4a27251 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:45.988217 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 810/1600] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:55:45 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:55:45.989290 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e8111de-471f-485c-bc4a-a1a0f4a27251 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:45.995212 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 791/1601] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:55:45 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:55:45.996478 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-141a80f7-6052-451f-86c2-c9ba0d5654ee None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:46.008068 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e8111de-471f-485c-bc4a-a1a0f4a27251 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:46.012002 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-141a80f7-6052-451f-86c2-c9ba0d5654ee None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:46.013972 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6362b0b-3fd2-48a2-8358-2037438eed9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:46.015031 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6362b0b-3fd2-48a2-8358-2037438eed9e admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:46.015427 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6362b0b-3fd2-48a2-8358-2037438eed9e admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:46.015820 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6362b0b-3fd2-48a2-8358-2037438eed9e admin admin] PATH_INFO: `/v3/projects/a7d8e82f58814da6b8c08bfa04955d4c/users/b027fb8adadc488e975c4541e7ee6626/roles/588016933bab436d8b7c3ac76827356d` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:46.016883 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec07a019-f2a7-4856-a5c3-fd7d6b7d287d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:46.018183 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec07a019-f2a7-4856-a5c3-fd7d6b7d287d admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:46.018470 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec07a019-f2a7-4856-a5c3-fd7d6b7d287d admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:46.018930 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec07a019-f2a7-4856-a5c3-fd7d6b7d287d admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:46.020472 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec07a019-f2a7-4856-a5c3-fd7d6b7d287d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:46.022875 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec07a019-f2a7-4856-a5c3-fd7d6b7d287d admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:46.033342 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6362b0b-3fd2-48a2-8358-2037438eed9e admin admin] RBAC: Authorizing `identity:create_grant(project_id=a7d8e82f58814da6b8c08bfa04955d4c, user_id=b027fb8adadc488e975c4541e7ee6626, role_id=588016933bab436d8b7c3ac76827356d)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:46.037896 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 792/1602] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:55:45 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:55:46.037896 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec07a019-f2a7-4856-a5c3-fd7d6b7d287d admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:46.038135 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6362b0b-3fd2-48a2-8358-2037438eed9e admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:46.047956 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec07a019-f2a7-4856-a5c3-fd7d6b7d287d admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:46.051488 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 811/1603] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:55:45 2022] PUT /identity/v3/projects/a7d8e82f58814da6b8c08bfa04955d4c/users/b027fb8adadc488e975c4541e7ee6626/roles/588016933bab436d8b7c3ac76827356d => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:55:46.052585 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6362b0b-3fd2-48a2-8358-2037438eed9e admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:46.056604 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-e6362b0b-3fd2-48a2-8358-2037438eed9e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:55:46.131418 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc71f59-c4af-4fdc-9a39-924fa1bf054d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:46.132275 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc71f59-c4af-4fdc-9a39-924fa1bf054d admin admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:46.132740 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc71f59-c4af-4fdc-9a39-924fa1bf054d admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:46.133187 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc71f59-c4af-4fdc-9a39-924fa1bf054d admin admin] PATH_INFO: `/v3/projects/b129a63b5ece4a578c2bcc961efe5060/users/9485c032a2d5462486ac73373db67a10/roles/588016933bab436d8b7c3ac76827356d` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:46.138632 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6362b0b-3fd2-48a2-8358-2037438eed9e admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:46.143894 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71a256b1-fbe0-4a9f-859b-297d301f73dd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:46.144670 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71a256b1-fbe0-4a9f-859b-297d301f73dd service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:46.145090 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71a256b1-fbe0-4a9f-859b-297d301f73dd service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:46.145420 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71a256b1-fbe0-4a9f-859b-297d301f73dd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:46.153169 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cc71f59-c4af-4fdc-9a39-924fa1bf054d admin admin] RBAC: Authorizing `identity:create_grant(project_id=b129a63b5ece4a578c2bcc961efe5060, user_id=9485c032a2d5462486ac73373db67a10, role_id=588016933bab436d8b7c3ac76827356d)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:46.155922 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71a256b1-fbe0-4a9f-859b-297d301f73dd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:46.157369 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71a256b1-fbe0-4a9f-859b-297d301f73dd service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:46.159206 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cc71f59-c4af-4fdc-9a39-924fa1bf054d admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:46.168641 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 812/1604] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:55:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 117 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:55:46.169781 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71a256b1-fbe0-4a9f-859b-297d301f73dd service neutron] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:46.180290 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71a256b1-fbe0-4a9f-859b-297d301f73dd service neutron] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:46.181588 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 793/1605] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:55:46 2022] PUT /identity/v3/projects/b129a63b5ece4a578c2bcc961efe5060/users/9485c032a2d5462486ac73373db67a10/roles/588016933bab436d8b7c3ac76827356d => generated 0 bytes in 146 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:55:46.182711 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc71f59-c4af-4fdc-9a39-924fa1bf054d admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:46.289218 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6c0fd04-e241-4cd8-937a-761dceb154d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:46.289952 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c0fd04-e241-4cd8-937a-761dceb154d6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:46.290422 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c0fd04-e241-4cd8-937a-761dceb154d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:46.290698 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c0fd04-e241-4cd8-937a-761dceb154d6 admin admin] PATH_INFO: `/v3/projects/197f7f9effc24c8eb190027e4d47cda3/users/8acc3f6ca4a84de99de98217f7d36d81/roles/588016933bab436d8b7c3ac76827356d` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:46.292448 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc71f59-c4af-4fdc-9a39-924fa1bf054d admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:46.310578 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be390e9c-bead-4a1c-9d0e-f2f96558ca2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:46.311252 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be390e9c-bead-4a1c-9d0e-f2f96558ca2e admin admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:46.311530 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be390e9c-bead-4a1c-9d0e-f2f96558ca2e admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:46.311844 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be390e9c-bead-4a1c-9d0e-f2f96558ca2e admin admin] PATH_INFO: `/v3/projects/f68941040b3049bd9c62e2fc061d94c1/users/438446d1e6b3402bb18b547c089391f8/roles/588016933bab436d8b7c3ac76827356d` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:46.313553 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6c0fd04-e241-4cd8-937a-761dceb154d6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=197f7f9effc24c8eb190027e4d47cda3, user_id=8acc3f6ca4a84de99de98217f7d36d81, role_id=588016933bab436d8b7c3ac76827356d)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:46.318851 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6c0fd04-e241-4cd8-937a-761dceb154d6 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:46.342684 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 813/1606] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:55:46 2022] PUT /identity/v3/projects/197f7f9effc24c8eb190027e4d47cda3/users/8acc3f6ca4a84de99de98217f7d36d81/roles/588016933bab436d8b7c3ac76827356d => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:55:46.343684 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6c0fd04-e241-4cd8-937a-761dceb154d6 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:46.353859 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be390e9c-bead-4a1c-9d0e-f2f96558ca2e admin admin] RBAC: Authorizing `identity:create_grant(project_id=f68941040b3049bd9c62e2fc061d94c1, user_id=438446d1e6b3402bb18b547c089391f8, role_id=588016933bab436d8b7c3ac76827356d)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:46.354145 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:55:46.356655 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be390e9c-bead-4a1c-9d0e-f2f96558ca2e admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:46.381586 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 794/1607] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:55:46 2022] PUT /identity/v3/projects/f68941040b3049bd9c62e2fc061d94c1/users/438446d1e6b3402bb18b547c089391f8/roles/588016933bab436d8b7c3ac76827356d => generated 0 bytes in 199 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:55:46.382716 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be390e9c-bead-4a1c-9d0e-f2f96558ca2e admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:46.394839 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:55:46.450816 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6c0fd04-e241-4cd8-937a-761dceb154d6 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:46.554095 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be390e9c-bead-4a1c-9d0e-f2f96558ca2e admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:46.568889 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc123860-069c-4381-a0e7-030a793945b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:46.569127 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cf34151-4411-4166-a478-cc063b4c39a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:46.569851 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf34151-4411-4166-a478-cc063b4c39a1 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:46.570196 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc123860-069c-4381-a0e7-030a793945b4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:46.570474 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf34151-4411-4166-a478-cc063b4c39a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:46.570838 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf34151-4411-4166-a478-cc063b4c39a1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:46.571021 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc123860-069c-4381-a0e7-030a793945b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:46.571576 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc123860-069c-4381-a0e7-030a793945b4 admin admin] PATH_INFO: `/v3/projects/071e8edf3cea479baab9f37f6dd378e3/users/87513018217c45a1be915892e301fc0b/roles/588016933bab436d8b7c3ac76827356d` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:46.572038 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cf34151-4411-4166-a478-cc063b4c39a1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:46.574924 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cf34151-4411-4166-a478-cc063b4c39a1 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:46.599114 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc123860-069c-4381-a0e7-030a793945b4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=071e8edf3cea479baab9f37f6dd378e3, user_id=87513018217c45a1be915892e301fc0b, role_id=588016933bab436d8b7c3ac76827356d)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:46.602335 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc123860-069c-4381-a0e7-030a793945b4 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:46.641676 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 795/1608] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:55:46 2022] PUT /identity/v3/projects/071e8edf3cea479baab9f37f6dd378e3/users/87513018217c45a1be915892e301fc0b/roles/588016933bab436d8b7c3ac76827356d => generated 0 bytes in 248 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:55:46.641676 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f5217d8-97b0-484d-b0d5-51eb24243730 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:46.641676 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f5217d8-97b0-484d-b0d5-51eb24243730 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:46.641676 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f5217d8-97b0-484d-b0d5-51eb24243730 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:46.648124 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 814/1609] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 00:55:46 2022] POST /identity/v3/projects => generated 363 bytes in 305 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:55:46.649546 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a217ae5-9ba3-48d0-aa9e-da095f81aa91 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:46.649869 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a217ae5-9ba3-48d0-aa9e-da095f81aa91 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:46.650147 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a217ae5-9ba3-48d0-aa9e-da095f81aa91 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:46.679614 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-9f5217d8-97b0-484d-b0d5-51eb24243730 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:55:46.682644 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-9f5217d8-97b0-484d-b0d5-51eb24243730 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:55:46.697859 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-5a217ae5-9ba3-48d0-aa9e-da095f81aa91 None None] MFA Rules not processed for user `9485c032a2d5462486ac73373db67a10`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:55:46.699102 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-5a217ae5-9ba3-48d0-aa9e-da095f81aa91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:55:46.816472 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 815/1610] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:55:46 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:55:46.822728 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-743b8128-8f55-4274-82d4-cd6834607854 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:46.822728 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-743b8128-8f55-4274-82d4-cd6834607854 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:46.822728 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-743b8128-8f55-4274-82d4-cd6834607854 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:46.828517 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 796/1611] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:55:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:55:46.831565 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-739e06ed-2a24-4fc1-a2a6-e4c8f67a33d3 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:46.831887 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-739e06ed-2a24-4fc1-a2a6-e4c8f67a33d3 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:46.832161 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-739e06ed-2a24-4fc1-a2a6-e4c8f67a33d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:46.917803 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-739e06ed-2a24-4fc1-a2a6-e4c8f67a33d3 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:55:46.917803 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-739e06ed-2a24-4fc1-a2a6-e4c8f67a33d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:55:46.917803 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 797/1612] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:55:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 57 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:55:46.919000 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-743b8128-8f55-4274-82d4-cd6834607854 None None] MFA Rules not processed for user `8acc3f6ca4a84de99de98217f7d36d81`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:55:46.919888 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-430bd95f-20a3-4851-b0d2-8efe4a107731 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:46.919888 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-430bd95f-20a3-4851-b0d2-8efe4a107731 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:46.919888 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-430bd95f-20a3-4851-b0d2-8efe4a107731 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:46.920110 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-743b8128-8f55-4274-82d4-cd6834607854 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:55:46.947697 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:55:46.960910 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-430bd95f-20a3-4851-b0d2-8efe4a107731 None None] MFA Rules not processed for user `87513018217c45a1be915892e301fc0b`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:55:46.963290 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-430bd95f-20a3-4851-b0d2-8efe4a107731 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:55:47.045571 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 816/1613] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:55:46 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 225 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:55:47.045571 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-743b8128-8f55-4274-82d4-cd6834607854 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:47.058237 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-743b8128-8f55-4274-82d4-cd6834607854 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:47.063856 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-053124b6-5043-4ef9-9a32-34ff11347f7d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:47.064589 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-053124b6-5043-4ef9-9a32-34ff11347f7d admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:47.064963 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-053124b6-5043-4ef9-9a32-34ff11347f7d admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:47.065303 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-053124b6-5043-4ef9-9a32-34ff11347f7d admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:47.066593 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-053124b6-5043-4ef9-9a32-34ff11347f7d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:47.069239 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-053124b6-5043-4ef9-9a32-34ff11347f7d admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:47.102620 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 798/1614] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:55:46 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 210 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:55:47.102620 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-430bd95f-20a3-4851-b0d2-8efe4a107731 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:47.111052 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 817/1615] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:55:47 2022] POST /identity/v3/users => generated 288 bytes in 68 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:55:47.112053 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-053124b6-5043-4ef9-9a32-34ff11347f7d admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:47.113761 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-430bd95f-20a3-4851-b0d2-8efe4a107731 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:47.121210 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-146a9d8d-7ca4-401e-9468-8489f99e3d43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:47.121828 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-146a9d8d-7ca4-401e-9468-8489f99e3d43 admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:47.122107 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-146a9d8d-7ca4-401e-9468-8489f99e3d43 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:47.122374 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-146a9d8d-7ca4-401e-9468-8489f99e3d43 admin admin] PATH_INFO: `/v3/projects` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:47.123003 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-146a9d8d-7ca4-401e-9468-8489f99e3d43 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:47.125075 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-146a9d8d-7ca4-401e-9468-8489f99e3d43 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:47.135828 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:55:47.170124 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 799/1616] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 00:55:47 2022] POST /identity/v3/projects => generated 373 bytes in 72 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:55:47.171253 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-146a9d8d-7ca4-401e-9468-8489f99e3d43 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:47.202184 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-053124b6-5043-4ef9-9a32-34ff11347f7d admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:47.280263 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-146a9d8d-7ca4-401e-9468-8489f99e3d43 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:47.282340 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dfa5f8d-eecf-4363-9330-c804d286e932 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:47.284659 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dfa5f8d-eecf-4363-9330-c804d286e932 service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:47.284659 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dfa5f8d-eecf-4363-9330-c804d286e932 service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:47.284659 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dfa5f8d-eecf-4363-9330-c804d286e932 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:47.289662 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f83a838-3ad9-4932-8965-8e546a88e6cb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:47.290355 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f83a838-3ad9-4932-8965-8e546a88e6cb service neutron] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:47.291823 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f83a838-3ad9-4932-8965-8e546a88e6cb service neutron] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:47.293081 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f83a838-3ad9-4932-8965-8e546a88e6cb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:47.393649 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dfa5f8d-eecf-4363-9330-c804d286e932 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:47.394054 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f83a838-3ad9-4932-8965-8e546a88e6cb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:47.395294 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dfa5f8d-eecf-4363-9330-c804d286e932 service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:47.395965 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f83a838-3ad9-4932-8965-8e546a88e6cb service neutron] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:47.408567 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 818/1617] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:55:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 297 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:55:47.409991 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 800/1618] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:55:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 239 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:55:47.410395 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bdffd47-d1d3-4e2c-95a8-3ddc43b8ae86 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:47.410686 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bdffd47-d1d3-4e2c-95a8-3ddc43b8ae86 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:47.410959 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bdffd47-d1d3-4e2c-95a8-3ddc43b8ae86 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:47.411068 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f83a838-3ad9-4932-8965-8e546a88e6cb service neutron] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:47.413242 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 819/1619] 10.222.0.33 () {54 vars in 961 bytes} [Sun Oct 9 00:55:47 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:55:47.414040 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bdffd47-d1d3-4e2c-95a8-3ddc43b8ae86 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:47.437006 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bdffd47-d1d3-4e2c-95a8-3ddc43b8ae86 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:47.448503 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8ce0739-ef52-464d-b9d0-d1452dfc1204 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:47.449110 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ce0739-ef52-464d-b9d0-d1452dfc1204 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:47.449388 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ce0739-ef52-464d-b9d0-d1452dfc1204 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:47.449658 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ce0739-ef52-464d-b9d0-d1452dfc1204 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:47.450367 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8ce0739-ef52-464d-b9d0-d1452dfc1204 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:47.452010 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8ce0739-ef52-464d-b9d0-d1452dfc1204 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:47.454105 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:55:47.468286 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 820/1620] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:55:47 2022] GET /identity/v3/roles => generated 1420 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:55:47.469279 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8ce0739-ef52-464d-b9d0-d1452dfc1204 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:47.486952 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8ce0739-ef52-464d-b9d0-d1452dfc1204 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:47.497750 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-273e43fe-4948-48f7-b208-603743f0c9ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:47.498420 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273e43fe-4948-48f7-b208-603743f0c9ac admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:47.498715 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273e43fe-4948-48f7-b208-603743f0c9ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:47.499015 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273e43fe-4948-48f7-b208-603743f0c9ac admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:47.499712 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-273e43fe-4948-48f7-b208-603743f0c9ac admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:47.503845 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-273e43fe-4948-48f7-b208-603743f0c9ac admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:47.521046 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f83a838-3ad9-4932-8965-8e546a88e6cb service neutron] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:47.528106 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4ceaf14-c022-42a7-af6e-a8d179d3fb3f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3f48335668a4e8f934ac11ff66cd308', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:47.530668 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4ceaf14-c022-42a7-af6e-a8d179d3fb3f service cinder] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:47.532220 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4ceaf14-c022-42a7-af6e-a8d179d3fb3f service cinder] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:47.532575 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4ceaf14-c022-42a7-af6e-a8d179d3fb3f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:47.558411 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 821/1621] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:55:47 2022] POST /identity/v3/users => generated 293 bytes in 86 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:55:47.560200 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd81f047-e7a3-482c-aba2-428ef6f778a7 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:47.560497 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd81f047-e7a3-482c-aba2-428ef6f778a7 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:47.560769 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd81f047-e7a3-482c-aba2-428ef6f778a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:47.655481 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-fd81f047-e7a3-482c-aba2-428ef6f778a7 None None] MFA Rules not processed for user `e0a5d95969e74157bb2d2a57207e1651`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:55:47.658699 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-fd81f047-e7a3-482c-aba2-428ef6f778a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:55:47.667131 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:55:47.672866 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4ceaf14-c022-42a7-af6e-a8d179d3fb3f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:47.677772 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4ceaf14-c022-42a7-af6e-a8d179d3fb3f service cinder] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:47.699027 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 801/1622] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:55:47 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 288 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:55:47.699922 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4ceaf14-c022-42a7-af6e-a8d179d3fb3f service cinder] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:47.725654 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4ceaf14-c022-42a7-af6e-a8d179d3fb3f service cinder] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:47.734609 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e34bf89-7140-4ad4-aeaa-c469e4b06438 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:47.735491 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e34bf89-7140-4ad4-aeaa-c469e4b06438 admin admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:47.735639 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e34bf89-7140-4ad4-aeaa-c469e4b06438 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:47.735889 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e34bf89-7140-4ad4-aeaa-c469e4b06438 admin admin] PATH_INFO: `/v3/projects/f83c1519130d44cc8930a375b6e026e3/users/c33bcc640c5f423d919bb1d1172adb8b/roles/092036ee46ac427baf4d56468575f697` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:47.770669 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e34bf89-7140-4ad4-aeaa-c469e4b06438 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f83c1519130d44cc8930a375b6e026e3, user_id=c33bcc640c5f423d919bb1d1172adb8b, role_id=092036ee46ac427baf4d56468575f697)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:47.786429 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e34bf89-7140-4ad4-aeaa-c469e4b06438 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:47.790149 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 822/1623] 10.222.0.33 () {58 vars in 1058 bytes} [Sun Oct 9 00:55:47 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 231 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:55:47.791334 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd81f047-e7a3-482c-aba2-428ef6f778a7 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:47.812269 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd81f047-e7a3-482c-aba2-428ef6f778a7 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:47.824843 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa264312-fcd6-461b-90d9-cd2a6bdc74e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:47.825535 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 802/1624] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:55:47 2022] PUT /identity/v3/projects/f83c1519130d44cc8930a375b6e026e3/users/c33bcc640c5f423d919bb1d1172adb8b/roles/092036ee46ac427baf4d56468575f697 => generated 0 bytes in 125 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:55:47.825653 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa264312-fcd6-461b-90d9-cd2a6bdc74e1 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:47.826084 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa264312-fcd6-461b-90d9-cd2a6bdc74e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:47.827148 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa264312-fcd6-461b-90d9-cd2a6bdc74e1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:47.829160 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa264312-fcd6-461b-90d9-cd2a6bdc74e1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:47.829450 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e34bf89-7140-4ad4-aeaa-c469e4b06438 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:47.835777 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa264312-fcd6-461b-90d9-cd2a6bdc74e1 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:47.862359 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 823/1625] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:55:47 2022] GET /identity/v3/roles => generated 1420 bytes in 71 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:55:47.864069 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa264312-fcd6-461b-90d9-cd2a6bdc74e1 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:47.890333 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa264312-fcd6-461b-90d9-cd2a6bdc74e1 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:47.965529 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e34bf89-7140-4ad4-aeaa-c469e4b06438 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:47.976272 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85335c8a-fd1a-469e-bfd0-74d93f5532ce service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a5d95969e74157bb2d2a57207e1651', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:47.977223 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85335c8a-fd1a-469e-bfd0-74d93f5532ce service nova] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:47.979527 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85335c8a-fd1a-469e-bfd0-74d93f5532ce service nova] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:47.980215 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85335c8a-fd1a-469e-bfd0-74d93f5532ce service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:48.049297 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5863ead-f47c-4133-9ffa-ad8c7f71e5fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:48.049970 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5863ead-f47c-4133-9ffa-ad8c7f71e5fd admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:48.050484 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5863ead-f47c-4133-9ffa-ad8c7f71e5fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:48.050484 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5863ead-f47c-4133-9ffa-ad8c7f71e5fd admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:48.051375 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5863ead-f47c-4133-9ffa-ad8c7f71e5fd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:48.061615 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5863ead-f47c-4133-9ffa-ad8c7f71e5fd admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:48.088271 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 824/1626] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:55:47 2022] GET /identity/v3/roles => generated 1420 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:55:48.092084 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5863ead-f47c-4133-9ffa-ad8c7f71e5fd admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:48.117247 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85335c8a-fd1a-469e-bfd0-74d93f5532ce service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:48.118857 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85335c8a-fd1a-469e-bfd0-74d93f5532ce service nova] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:48.129314 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5863ead-f47c-4133-9ffa-ad8c7f71e5fd admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:48.146413 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff679838-d35a-4161-991c-c3df80fd39d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:48.148919 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 803/1627] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:55:47 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 324 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:55:48.149541 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85335c8a-fd1a-469e-bfd0-74d93f5532ce service nova] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:48.150466 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff679838-d35a-4161-991c-c3df80fd39d6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:48.150743 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff679838-d35a-4161-991c-c3df80fd39d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:48.151015 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff679838-d35a-4161-991c-c3df80fd39d6 admin admin] PATH_INFO: `/v3/projects/9201690b2e83499ea2a735e6c673b8af/users/098962dfac444426a5d60f7fca1f3efd/roles/092036ee46ac427baf4d56468575f697` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:48.188884 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85335c8a-fd1a-469e-bfd0-74d93f5532ce service nova] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:48.194688 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff679838-d35a-4161-991c-c3df80fd39d6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9201690b2e83499ea2a735e6c673b8af, user_id=098962dfac444426a5d60f7fca1f3efd, role_id=092036ee46ac427baf4d56468575f697)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:48.198799 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff679838-d35a-4161-991c-c3df80fd39d6 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:48.202653 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-470783ef-d57e-4b92-81c1-cac4ba661438 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:48.203638 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-470783ef-d57e-4b92-81c1-cac4ba661438 admin admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:48.203776 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-470783ef-d57e-4b92-81c1-cac4ba661438 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:48.204021 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-470783ef-d57e-4b92-81c1-cac4ba661438 admin admin] PATH_INFO: `/v3/projects/f83c1519130d44cc8930a375b6e026e3/users/c33bcc640c5f423d919bb1d1172adb8b/roles/092036ee46ac427baf4d56468575f697` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:48.208418 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-470783ef-d57e-4b92-81c1-cac4ba661438 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f83c1519130d44cc8930a375b6e026e3, user_id=c33bcc640c5f423d919bb1d1172adb8b, role_id=092036ee46ac427baf4d56468575f697)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:48.211315 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-470783ef-d57e-4b92-81c1-cac4ba661438 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:48.233643 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 825/1628] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:55:48 2022] PUT /identity/v3/projects/9201690b2e83499ea2a735e6c673b8af/users/098962dfac444426a5d60f7fca1f3efd/roles/092036ee46ac427baf4d56468575f697 => generated 0 bytes in 145 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:55:48.239337 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff679838-d35a-4161-991c-c3df80fd39d6 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:48.249470 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 804/1629] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:55:48 2022] PUT /identity/v3/projects/f83c1519130d44cc8930a375b6e026e3/users/c33bcc640c5f423d919bb1d1172adb8b/roles/092036ee46ac427baf4d56468575f697 => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:55:48.260247 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c984407f-d8c6-46f5-94b0-83341083c84b None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:48.260533 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c984407f-d8c6-46f5-94b0-83341083c84b None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:48.260803 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c984407f-d8c6-46f5-94b0-83341083c84b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:48.301724 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:55:48.348880 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-c984407f-d8c6-46f5-94b0-83341083c84b None None] MFA Rules not processed for user `c33bcc640c5f423d919bb1d1172adb8b`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:55:48.350150 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-c984407f-d8c6-46f5-94b0-83341083c84b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:55:48.442880 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:55:48.465973 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff679838-d35a-4161-991c-c3df80fd39d6 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:48.526170 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 805/1630] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:55:48 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 263 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:55:48.535683 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c984407f-d8c6-46f5-94b0-83341083c84b None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:48.591408 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab49bbec-58f1-4a90-bcb3-578feb18d9ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:48.592201 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab49bbec-58f1-4a90-bcb3-578feb18d9ad admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:48.592579 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab49bbec-58f1-4a90-bcb3-578feb18d9ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:48.592975 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab49bbec-58f1-4a90-bcb3-578feb18d9ad admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:48.593722 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab49bbec-58f1-4a90-bcb3-578feb18d9ad admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:48.596063 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab49bbec-58f1-4a90-bcb3-578feb18d9ad admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:48.637083 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 826/1631] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:55:48 2022] GET /identity/v3/roles => generated 1420 bytes in 398 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:55:48.638097 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab49bbec-58f1-4a90-bcb3-578feb18d9ad admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:48.662285 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab49bbec-58f1-4a90-bcb3-578feb18d9ad admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:48.674792 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63af0d16-7638-4332-95cb-aea89aba1b23 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:48.675722 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63af0d16-7638-4332-95cb-aea89aba1b23 admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:48.676107 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63af0d16-7638-4332-95cb-aea89aba1b23 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:48.677086 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63af0d16-7638-4332-95cb-aea89aba1b23 admin admin] PATH_INFO: `/v3/projects/9201690b2e83499ea2a735e6c673b8af/users/098962dfac444426a5d60f7fca1f3efd/roles/092036ee46ac427baf4d56468575f697` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:48.680154 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63af0d16-7638-4332-95cb-aea89aba1b23 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9201690b2e83499ea2a735e6c673b8af, user_id=098962dfac444426a5d60f7fca1f3efd, role_id=092036ee46ac427baf4d56468575f697)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:48.680718 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c984407f-d8c6-46f5-94b0-83341083c84b None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:48.683296 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63af0d16-7638-4332-95cb-aea89aba1b23 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:48.694947 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5848d9f0-9d4c-459c-b518-75c3beea900d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3f48335668a4e8f934ac11ff66cd308', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:48.695891 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5848d9f0-9d4c-459c-b518-75c3beea900d service cinder] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:48.696343 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5848d9f0-9d4c-459c-b518-75c3beea900d service cinder] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:48.696774 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5848d9f0-9d4c-459c-b518-75c3beea900d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:48.721449 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 827/1632] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:55:48 2022] PUT /identity/v3/projects/9201690b2e83499ea2a735e6c673b8af/users/098962dfac444426a5d60f7fca1f3efd/roles/092036ee46ac427baf4d56468575f697 => generated 0 bytes in 72 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:55:48.721449 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2feac1f2-c5de-441b-b6de-3670863411f3 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:48.721449 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2feac1f2-c5de-441b-b6de-3670863411f3 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:48.721449 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2feac1f2-c5de-441b-b6de-3670863411f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:48.723650 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5848d9f0-9d4c-459c-b518-75c3beea900d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:48.725503 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5848d9f0-9d4c-459c-b518-75c3beea900d service cinder] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:48.766396 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 806/1633] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:55:48 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 231 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 09 00:55:48.780286 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-650d2586-5592-481a-ac4e-7cd50b40be3b None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:48.780591 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-650d2586-5592-481a-ac4e-7cd50b40be3b None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:48.780840 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-650d2586-5592-481a-ac4e-7cd50b40be3b None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:48.788882 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 807/1634] 10.222.0.33 () {54 vars in 912 bytes} [Sun Oct 9 00:55:48 2022] GET /identity => generated 270 bytes in 22 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:55:48.789716 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-2feac1f2-c5de-441b-b6de-3670863411f3 None None] MFA Rules not processed for user `098962dfac444426a5d60f7fca1f3efd`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:55:48.791237 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90537cab-d044-4634-8677-648d7bfe7d40 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:48.793415 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-2feac1f2-c5de-441b-b6de-3670863411f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:55:48.794387 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90537cab-d044-4634-8677-648d7bfe7d40 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:48.797636 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90537cab-d044-4634-8677-648d7bfe7d40 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:48.864390 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-90537cab-d044-4634-8677-648d7bfe7d40 None None] MFA Rules not processed for user `e0a5d95969e74157bb2d2a57207e1651`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:55:48.869552 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-90537cab-d044-4634-8677-648d7bfe7d40 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:55:49.046103 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 828/1635] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:55:48 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 326 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:55:49.093319 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2feac1f2-c5de-441b-b6de-3670863411f3 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:49.136323 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 808/1636] 10.222.0.33 () {58 vars in 1009 bytes} [Sun Oct 9 00:55:48 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 347 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:55:49.231170 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:55:49.276646 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90537cab-d044-4634-8677-648d7bfe7d40 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:49.304313 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2feac1f2-c5de-441b-b6de-3670863411f3 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:49.317094 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b5b81f0-de1c-4966-b23b-e2b815f84e1f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3f48335668a4e8f934ac11ff66cd308', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:49.317730 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b5b81f0-de1c-4966-b23b-e2b815f84e1f service cinder] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:49.318015 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b5b81f0-de1c-4966-b23b-e2b815f84e1f service cinder] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:49.319822 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b5b81f0-de1c-4966-b23b-e2b815f84e1f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:49.336106 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b5b81f0-de1c-4966-b23b-e2b815f84e1f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:49.337788 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b5b81f0-de1c-4966-b23b-e2b815f84e1f service cinder] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:49.365675 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 829/1637] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:55:49 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 273 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:55:49.383531 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90537cab-d044-4634-8677-648d7bfe7d40 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:49.390360 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e4066b3-4456-4336-9c2f-937bd666fd43 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3f48335668a4e8f934ac11ff66cd308', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:49.391070 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e4066b3-4456-4336-9c2f-937bd666fd43 service cinder] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:49.391405 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e4066b3-4456-4336-9c2f-937bd666fd43 service cinder] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:49.391803 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e4066b3-4456-4336-9c2f-937bd666fd43 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:49.409019 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e4066b3-4456-4336-9c2f-937bd666fd43 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:49.410803 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e4066b3-4456-4336-9c2f-937bd666fd43 service cinder] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:49.426819 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 809/1638] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:55:49 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 151 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:55:49.694867 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b5b81f0-de1c-4966-b23b-e2b815f84e1f service cinder] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:49.719969 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b5b81f0-de1c-4966-b23b-e2b815f84e1f service cinder] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:49.727528 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8177f78-27df-449c-88aa-f50879100b17 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3f48335668a4e8f934ac11ff66cd308', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:49.728181 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8177f78-27df-449c-88aa-f50879100b17 service cinder] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:49.728469 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8177f78-27df-449c-88aa-f50879100b17 service cinder] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:49.728745 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8177f78-27df-449c-88aa-f50879100b17 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:49.906618 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8177f78-27df-449c-88aa-f50879100b17 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:49.908842 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8177f78-27df-449c-88aa-f50879100b17 service cinder] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:49.932940 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 830/1639] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:55:49 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 238 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:55:50.787035 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e7fd59a-fa5e-4ed1-93b4-845b428b370e None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:50.787925 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e7fd59a-fa5e-4ed1-93b4-845b428b370e None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:50.788351 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e7fd59a-fa5e-4ed1-93b4-845b428b370e None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:50.790179 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 810/1640] 10.222.0.33 () {54 vars in 961 bytes} [Sun Oct 9 00:55:50 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:55:50.813866 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c990e92b-1fd3-40a9-a02b-5600fb7ace24 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:50.814497 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c990e92b-1fd3-40a9-a02b-5600fb7ace24 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:50.814497 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c990e92b-1fd3-40a9-a02b-5600fb7ace24 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:50.866574 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-c990e92b-1fd3-40a9-a02b-5600fb7ace24 None None] MFA Rules not processed for user `e0a5d95969e74157bb2d2a57207e1651`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:55:50.876913 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-c990e92b-1fd3-40a9-a02b-5600fb7ace24 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:55:50.902089 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 831/1641] 10.222.0.33 () {58 vars in 1058 bytes} [Sun Oct 9 00:55:50 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:55:50.912078 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c3c266e-5e7f-4b85-b272-cb2cb55a4fc5 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:50.912247 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c3c266e-5e7f-4b85-b272-cb2cb55a4fc5 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:50.912502 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c3c266e-5e7f-4b85-b272-cb2cb55a4fc5 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:50.914284 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 811/1642] 10.222.0.33 () {54 vars in 912 bytes} [Sun Oct 9 00:55:50 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:55:50.921290 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-647cde70-caeb-43d9-a09a-4536cab93d1d None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:50.921663 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-647cde70-caeb-43d9-a09a-4536cab93d1d None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:50.922032 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-647cde70-caeb-43d9-a09a-4536cab93d1d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:50.968083 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-647cde70-caeb-43d9-a09a-4536cab93d1d None None] MFA Rules not processed for user `e0a5d95969e74157bb2d2a57207e1651`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:55:50.972983 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-647cde70-caeb-43d9-a09a-4536cab93d1d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:55:51.012485 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 832/1643] 10.222.0.33 () {58 vars in 1009 bytes} [Sun Oct 9 00:55:50 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:55:51.072654 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c3c266e-5e7f-4b85-b272-cb2cb55a4fc5 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:51.089979 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c3c266e-5e7f-4b85-b272-cb2cb55a4fc5 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:51.096713 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-416f6057-6aa1-49da-bb12-ae180fd88738 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3f48335668a4e8f934ac11ff66cd308', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:51.097659 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-416f6057-6aa1-49da-bb12-ae180fd88738 service cinder] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:51.098141 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-416f6057-6aa1-49da-bb12-ae180fd88738 service cinder] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:51.099091 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-416f6057-6aa1-49da-bb12-ae180fd88738 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:51.116599 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-416f6057-6aa1-49da-bb12-ae180fd88738 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:51.118455 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-416f6057-6aa1-49da-bb12-ae180fd88738 service cinder] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:51.134541 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 812/1644] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:55:51 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 62 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:55:51.758094 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-647cde70-caeb-43d9-a09a-4536cab93d1d None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:51.966429 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-647cde70-caeb-43d9-a09a-4536cab93d1d None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:51.983816 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dcd36dc-6bfc-429e-bf65-3e2e597caf7d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:51.985322 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dcd36dc-6bfc-429e-bf65-3e2e597caf7d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:51.987734 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dcd36dc-6bfc-429e-bf65-3e2e597caf7d admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:51.989103 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dcd36dc-6bfc-429e-bf65-3e2e597caf7d admin admin] PATH_INFO: `/v3/users/f25b1b172aff4376b6c6905cb3b12913` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:51.994320 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dcd36dc-6bfc-429e-bf65-3e2e597caf7d admin admin] RBAC: Authorizing `identity:delete_user(user_id=f25b1b172aff4376b6c6905cb3b12913)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:51.995710 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: WARNING py.warnings [None req-1dcd36dc-6bfc-429e-bf65-3e2e597caf7d admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 09 00:55:51.995710 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: warnings.warn(msg) Oct 09 00:55:51.995710 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: Oct 09 00:55:51.997325 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dcd36dc-6bfc-429e-bf65-3e2e597caf7d admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:52.151371 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-1dcd36dc-6bfc-429e-bf65-3e2e597caf7d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f25b1b172aff4376b6c6905cb3b12913'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:55:52.186347 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-1dcd36dc-6bfc-429e-bf65-3e2e597caf7d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f25b1b172aff4376b6c6905cb3b12913'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:55:52.188710 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-1dcd36dc-6bfc-429e-bf65-3e2e597caf7d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f25b1b172aff4376b6c6905cb3b12913'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:55:52.192326 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-1dcd36dc-6bfc-429e-bf65-3e2e597caf7d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f25b1b172aff4376b6c6905cb3b12913'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:55:52.215656 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-1dcd36dc-6bfc-429e-bf65-3e2e597caf7d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f25b1b172aff4376b6c6905cb3b12913'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:55:52.217879 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:55:52.287506 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 833/1645] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 00:55:51 2022] DELETE /identity/v3/users/f25b1b172aff4376b6c6905cb3b12913 => generated 0 bytes in 530 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:55:52.291500 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed789150-9ec7-4cd0-82c7-2240e6465795 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:52.291950 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed789150-9ec7-4cd0-82c7-2240e6465795 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:52.292112 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed789150-9ec7-4cd0-82c7-2240e6465795 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:52.305048 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:55:52.359346 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-ed789150-9ec7-4cd0-82c7-2240e6465795 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:55:52.363130 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-ed789150-9ec7-4cd0-82c7-2240e6465795 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:55:52.533977 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 813/1646] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:55:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 243 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:55:52.614693 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dcd36dc-6bfc-429e-bf65-3e2e597caf7d admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:52.616412 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-1dcd36dc-6bfc-429e-bf65-3e2e597caf7d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:55:52.706525 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dcd36dc-6bfc-429e-bf65-3e2e597caf7d admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:52.713903 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9068413a-72c9-4748-aa40-aad3cc95246a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:52.715104 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9068413a-72c9-4748-aa40-aad3cc95246a service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:52.715414 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9068413a-72c9-4748-aa40-aad3cc95246a service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:52.715832 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9068413a-72c9-4748-aa40-aad3cc95246a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:52.738434 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9068413a-72c9-4748-aa40-aad3cc95246a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:52.740759 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9068413a-72c9-4748-aa40-aad3cc95246a service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:52.763705 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 834/1647] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:55:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 149 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:55:54.524624 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed789150-9ec7-4cd0-82c7-2240e6465795 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:54.526883 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-ed789150-9ec7-4cd0-82c7-2240e6465795 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:55:54.544908 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed789150-9ec7-4cd0-82c7-2240e6465795 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:54.557685 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10d7df1d-66b3-4750-89a0-117038abe5ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:54.558793 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d7df1d-66b3-4750-89a0-117038abe5ae admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:54.559093 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d7df1d-66b3-4750-89a0-117038abe5ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:54.559346 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d7df1d-66b3-4750-89a0-117038abe5ae admin admin] PATH_INFO: `/v3/projects/0ef954aff8eb4042b4fedfc47c6d073e` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:54.560905 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10d7df1d-66b3-4750-89a0-117038abe5ae admin admin] RBAC: Authorizing `identity:delete_project(project_id=0ef954aff8eb4042b4fedfc47c6d073e)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:54.562604 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: WARNING py.warnings [None req-10d7df1d-66b3-4750-89a0-117038abe5ae admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 09 00:55:54.562604 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: warnings.warn(msg) Oct 09 00:55:54.562604 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: Oct 09 00:55:54.565047 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10d7df1d-66b3-4750-89a0-117038abe5ae admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:54.615626 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac4b5b5-1bee-4b3d-a9c5-31cf76f38b95 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:54.616171 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac4b5b5-1bee-4b3d-a9c5-31cf76f38b95 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:54.616171 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac4b5b5-1bee-4b3d-a9c5-31cf76f38b95 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:54.643141 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-10d7df1d-66b3-4750-89a0-117038abe5ae admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0ef954aff8eb4042b4fedfc47c6d073e', 'request_id': 'req-10d7df1d-66b3-4750-89a0-117038abe5ae'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:55:54.650774 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-10d7df1d-66b3-4750-89a0-117038abe5ae admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0ef954aff8eb4042b4fedfc47c6d073e', 'request_id': 'req-10d7df1d-66b3-4750-89a0-117038abe5ae'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:55:54.669852 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-10d7df1d-66b3-4750-89a0-117038abe5ae admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0ef954aff8eb4042b4fedfc47c6d073e', 'request_id': 'req-10d7df1d-66b3-4750-89a0-117038abe5ae'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:55:54.684236 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-6ac4b5b5-1bee-4b3d-a9c5-31cf76f38b95 None None] MFA Rules not processed for user `1512c91201f14b75a9be35b89c2097ef`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:55:54.686132 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-6ac4b5b5-1bee-4b3d-a9c5-31cf76f38b95 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:55:54.693817 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-10d7df1d-66b3-4750-89a0-117038abe5ae admin admin] The token cache is being invalidate because project 0ef954aff8eb4042b4fedfc47c6d073e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81453) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 00:55:54.694279 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-10d7df1d-66b3-4750-89a0-117038abe5ae admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:55:54.695050 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-10d7df1d-66b3-4750-89a0-117038abe5ae admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:55:54.706285 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 814/1648] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 00:55:54 2022] DELETE /identity/v3/projects/0ef954aff8eb4042b4fedfc47c6d073e => generated 0 bytes in 182 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:55:54.710985 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10d7df1d-66b3-4750-89a0-117038abe5ae admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:54.731443 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-10d7df1d-66b3-4750-89a0-117038abe5ae admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:55:54.860820 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 835/1649] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:55:54 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 245 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 09 00:55:54.879277 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ac4b5b5-1bee-4b3d-a9c5-31cf76f38b95 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:54.880908 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-6ac4b5b5-1bee-4b3d-a9c5-31cf76f38b95 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:55:54.929182 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10d7df1d-66b3-4750-89a0-117038abe5ae admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:54.934323 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ed0c4b3-d4a3-46f0-8cca-36784d4bc8bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:54.935091 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed0c4b3-d4a3-46f0-8cca-36784d4bc8bf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:54.935575 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed0c4b3-d4a3-46f0-8cca-36784d4bc8bf admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:54.935976 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed0c4b3-d4a3-46f0-8cca-36784d4bc8bf admin admin] PATH_INFO: `/v3/users/c33bcc640c5f423d919bb1d1172adb8b` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:54.938356 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ed0c4b3-d4a3-46f0-8cca-36784d4bc8bf admin admin] RBAC: Authorizing `identity:delete_user(user_id=c33bcc640c5f423d919bb1d1172adb8b)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:54.941949 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ed0c4b3-d4a3-46f0-8cca-36784d4bc8bf admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:54.989923 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:55:55.023105 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-8ed0c4b3-d4a3-46f0-8cca-36784d4bc8bf admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c33bcc640c5f423d919bb1d1172adb8b'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:55:55.031776 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-8ed0c4b3-d4a3-46f0-8cca-36784d4bc8bf admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c33bcc640c5f423d919bb1d1172adb8b'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:55:55.032562 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-8ed0c4b3-d4a3-46f0-8cca-36784d4bc8bf admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c33bcc640c5f423d919bb1d1172adb8b'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:55:55.047082 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-8ed0c4b3-d4a3-46f0-8cca-36784d4bc8bf admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c33bcc640c5f423d919bb1d1172adb8b'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:55:55.050503 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ac4b5b5-1bee-4b3d-a9c5-31cf76f38b95 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:55.071529 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2cf9a57-9148-4d95-af05-439f89d5e0be service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a5d95969e74157bb2d2a57207e1651', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:55.074983 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2cf9a57-9148-4d95-af05-439f89d5e0be service nova] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:55.075747 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2cf9a57-9148-4d95-af05-439f89d5e0be service nova] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:55.089705 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2cf9a57-9148-4d95-af05-439f89d5e0be service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:55.093740 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-c2cf9a57-9148-4d95-af05-439f89d5e0be service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:55:55.115423 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-8ed0c4b3-d4a3-46f0-8cca-36784d4bc8bf admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c33bcc640c5f423d919bb1d1172adb8b'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:55:55.132258 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 815/1650] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 00:55:54 2022] DELETE /identity/v3/users/c33bcc640c5f423d919bb1d1172adb8b => generated 0 bytes in 422 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:55:55.136196 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2cf9a57-9148-4d95-af05-439f89d5e0be service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:55.139008 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2cf9a57-9148-4d95-af05-439f89d5e0be service nova] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:55.145850 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:55:55.154690 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 836/1651] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:55:54 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 276 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:55:55.649704 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ed0c4b3-d4a3-46f0-8cca-36784d4bc8bf admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:55.652387 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-8ed0c4b3-d4a3-46f0-8cca-36784d4bc8bf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:55:55.841577 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ed0c4b3-d4a3-46f0-8cca-36784d4bc8bf admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:55.848830 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75a33072-f252-4ed5-b2ab-2664d2db1f65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:55.851767 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75a33072-f252-4ed5-b2ab-2664d2db1f65 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:55.852240 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75a33072-f252-4ed5-b2ab-2664d2db1f65 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:55.852938 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75a33072-f252-4ed5-b2ab-2664d2db1f65 admin admin] PATH_INFO: `/v3/projects/f83c1519130d44cc8930a375b6e026e3` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:55.855085 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75a33072-f252-4ed5-b2ab-2664d2db1f65 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f83c1519130d44cc8930a375b6e026e3)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:55.858422 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75a33072-f252-4ed5-b2ab-2664d2db1f65 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:55.994770 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-75a33072-f252-4ed5-b2ab-2664d2db1f65 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f83c1519130d44cc8930a375b6e026e3', 'request_id': 'req-75a33072-f252-4ed5-b2ab-2664d2db1f65'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:55:56.004039 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-75a33072-f252-4ed5-b2ab-2664d2db1f65 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f83c1519130d44cc8930a375b6e026e3', 'request_id': 'req-75a33072-f252-4ed5-b2ab-2664d2db1f65'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:55:56.017675 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-75a33072-f252-4ed5-b2ab-2664d2db1f65 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f83c1519130d44cc8930a375b6e026e3', 'request_id': 'req-75a33072-f252-4ed5-b2ab-2664d2db1f65'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:55:56.073482 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-75a33072-f252-4ed5-b2ab-2664d2db1f65 admin admin] The token cache is being invalidate because project f83c1519130d44cc8930a375b6e026e3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81453) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 00:55:56.073961 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-75a33072-f252-4ed5-b2ab-2664d2db1f65 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:55:56.075037 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-75a33072-f252-4ed5-b2ab-2664d2db1f65 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:55:56.080954 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 816/1652] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 00:55:55 2022] DELETE /identity/v3/projects/f83c1519130d44cc8930a375b6e026e3 => generated 0 bytes in 434 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:55:56.091348 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b90c68-236e-488d-bd13-b3c97e689cd1 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:56.091825 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b90c68-236e-488d-bd13-b3c97e689cd1 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:56.091951 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b90c68-236e-488d-bd13-b3c97e689cd1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:56.132771 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-c7b90c68-236e-488d-bd13-b3c97e689cd1 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:55:56.134754 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-c7b90c68-236e-488d-bd13-b3c97e689cd1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:55:56.265420 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b786fdf-15c2-427b-900f-5dddf9bd8797 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:56.266087 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b786fdf-15c2-427b-900f-5dddf9bd8797 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:56.266551 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b786fdf-15c2-427b-900f-5dddf9bd8797 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:56.276052 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 837/1653] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:55:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:55:56.306881 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-2b786fdf-15c2-427b-900f-5dddf9bd8797 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:55:56.316076 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-2b786fdf-15c2-427b-900f-5dddf9bd8797 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:55:56.336988 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7b90c68-236e-488d-bd13-b3c97e689cd1 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:56.397633 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 817/1654] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:55:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 75 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:55:56.409925 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7b90c68-236e-488d-bd13-b3c97e689cd1 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:56.434997 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65292529-926a-4d2b-8336-6aaa1f28f0e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:56.442750 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65292529-926a-4d2b-8336-6aaa1f28f0e6 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:56.443458 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65292529-926a-4d2b-8336-6aaa1f28f0e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:56.443942 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65292529-926a-4d2b-8336-6aaa1f28f0e6 admin admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:56.448419 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65292529-926a-4d2b-8336-6aaa1f28f0e6 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:55:56.449434 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65292529-926a-4d2b-8336-6aaa1f28f0e6 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:56.457026 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65292529-926a-4d2b-8336-6aaa1f28f0e6 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:56.527968 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 838/1655] 10.222.0.33 () {58 vars in 1153 bytes} [Sun Oct 9 00:55:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:55:56.528523 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b786fdf-15c2-427b-900f-5dddf9bd8797 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:56.540241 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b786fdf-15c2-427b-900f-5dddf9bd8797 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:56.557614 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63017538-dc62-43a7-b85a-e97d4553c967 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:56.558382 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63017538-dc62-43a7-b85a-e97d4553c967 admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:56.558693 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63017538-dc62-43a7-b85a-e97d4553c967 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:56.559034 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63017538-dc62-43a7-b85a-e97d4553c967 admin admin] PATH_INFO: `/v3/projects` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:56.560006 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63017538-dc62-43a7-b85a-e97d4553c967 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:56.567648 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63017538-dc62-43a7-b85a-e97d4553c967 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:56.657454 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 818/1656] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 00:55:56 2022] POST /identity/v3/projects => generated 367 bytes in 129 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:55:56.661716 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65292529-926a-4d2b-8336-6aaa1f28f0e6 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:56.681021 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65292529-926a-4d2b-8336-6aaa1f28f0e6 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:56.928645 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da7a269-0c55-4ba1-9386-2f70f8362bef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:56.929286 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da7a269-0c55-4ba1-9386-2f70f8362bef admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:56.929594 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da7a269-0c55-4ba1-9386-2f70f8362bef admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:56.929902 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da7a269-0c55-4ba1-9386-2f70f8362bef admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:56.933445 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6da7a269-0c55-4ba1-9386-2f70f8362bef admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:56.940560 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6da7a269-0c55-4ba1-9386-2f70f8362bef admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:56.986748 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 839/1657] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:55:56 2022] POST /identity/v3/users => generated 284 bytes in 326 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:55:56.990959 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63017538-dc62-43a7-b85a-e97d4553c967 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:57.008656 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63017538-dc62-43a7-b85a-e97d4553c967 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:57.025707 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e2b0fc3-6718-4a78-9520-5ed348c63991 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:57.026598 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e2b0fc3-6718-4a78-9520-5ed348c63991 admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:57.027726 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e2b0fc3-6718-4a78-9520-5ed348c63991 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:57.028652 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e2b0fc3-6718-4a78-9520-5ed348c63991 admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:57.030147 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e2b0fc3-6718-4a78-9520-5ed348c63991 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:57.038098 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e2b0fc3-6718-4a78-9520-5ed348c63991 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:57.042492 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=85,keystone:DELETE=12,keystone:INSERT=3 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:55:57.067777 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 819/1658] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:55:56 2022] GET /identity/v3/roles => generated 1420 bytes in 77 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:55:57.084979 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da7a269-0c55-4ba1-9386-2f70f8362bef admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:57.097842 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da7a269-0c55-4ba1-9386-2f70f8362bef admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:57.104568 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e99b522e-080c-4f0b-ac3c-98ea173a2239 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:57.105751 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99b522e-080c-4f0b-ac3c-98ea173a2239 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:57.106250 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99b522e-080c-4f0b-ac3c-98ea173a2239 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:57.106729 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99b522e-080c-4f0b-ac3c-98ea173a2239 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:57.107658 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e99b522e-080c-4f0b-ac3c-98ea173a2239 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:57.109613 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e99b522e-080c-4f0b-ac3c-98ea173a2239 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:57.140243 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 840/1659] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:55:57 2022] GET /identity/v3/roles => generated 1420 bytes in 69 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:55:57.144968 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e2b0fc3-6718-4a78-9520-5ed348c63991 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:57.162866 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e2b0fc3-6718-4a78-9520-5ed348c63991 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:57.186677 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7188bdc4-4f73-409b-9fe2-3cbb00a22115 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:57.192132 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7188bdc4-4f73-409b-9fe2-3cbb00a22115 admin admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:57.192644 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7188bdc4-4f73-409b-9fe2-3cbb00a22115 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:57.193059 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7188bdc4-4f73-409b-9fe2-3cbb00a22115 admin admin] PATH_INFO: `/v3/projects/c283e71634c340be94a2248f9a5f9b82/users/aaf45940259542d7bea290b45ac9db64/roles/588016933bab436d8b7c3ac76827356d` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:57.233012 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7188bdc4-4f73-409b-9fe2-3cbb00a22115 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c283e71634c340be94a2248f9a5f9b82, user_id=aaf45940259542d7bea290b45ac9db64, role_id=588016933bab436d8b7c3ac76827356d)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:57.238126 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7188bdc4-4f73-409b-9fe2-3cbb00a22115 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:57.280942 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 820/1660] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:55:57 2022] PUT /identity/v3/projects/c283e71634c340be94a2248f9a5f9b82/users/aaf45940259542d7bea290b45ac9db64/roles/588016933bab436d8b7c3ac76827356d => generated 0 bytes in 137 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:55:57.294662 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2280a76d-0837-426e-af0b-ea479bdebb1a None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:57.295324 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2280a76d-0837-426e-af0b-ea479bdebb1a None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:57.296150 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2280a76d-0837-426e-af0b-ea479bdebb1a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:57.376126 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-2280a76d-0837-426e-af0b-ea479bdebb1a None None] MFA Rules not processed for user `aaf45940259542d7bea290b45ac9db64`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:55:57.378379 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-2280a76d-0837-426e-af0b-ea479bdebb1a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:55:57.562135 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:55:57.611427 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 841/1661] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:55:57 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 316 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:55:57.612914 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7188bdc4-4f73-409b-9fe2-3cbb00a22115 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:57.816883 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7188bdc4-4f73-409b-9fe2-3cbb00a22115 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:57.838971 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7bbf94c-0658-46c8-8049-f00f9cc479a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:57.838971 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7bbf94c-0658-46c8-8049-f00f9cc479a8 admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:57.839680 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7bbf94c-0658-46c8-8049-f00f9cc479a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:57.840122 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7bbf94c-0658-46c8-8049-f00f9cc479a8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:57.840922 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7bbf94c-0658-46c8-8049-f00f9cc479a8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:57.842986 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7bbf94c-0658-46c8-8049-f00f9cc479a8 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:57.916011 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 821/1662] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 00:55:57 2022] POST /identity/v3/projects => generated 367 bytes in 303 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:55:57.916933 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2280a76d-0837-426e-af0b-ea479bdebb1a None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:57.936751 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2280a76d-0837-426e-af0b-ea479bdebb1a None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:58.070640 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0078db31-da11-43c8-b41b-777b0d5b131d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:58.071267 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0078db31-da11-43c8-b41b-777b0d5b131d admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:58.071547 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0078db31-da11-43c8-b41b-777b0d5b131d admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:58.071865 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0078db31-da11-43c8-b41b-777b0d5b131d admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:58.087165 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0078db31-da11-43c8-b41b-777b0d5b131d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:58.087165 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0078db31-da11-43c8-b41b-777b0d5b131d admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:58.136490 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 842/1663] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:55:57 2022] POST /identity/v3/users => generated 290 bytes in 220 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 09 00:55:58.141262 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7bbf94c-0658-46c8-8049-f00f9cc479a8 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:58.166102 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7bbf94c-0658-46c8-8049-f00f9cc479a8 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:58.184668 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef222d53-8942-4c7b-9223-89297ae6b9c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:58.186346 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef222d53-8942-4c7b-9223-89297ae6b9c9 admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:58.186346 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef222d53-8942-4c7b-9223-89297ae6b9c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:58.186346 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef222d53-8942-4c7b-9223-89297ae6b9c9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:58.186563 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef222d53-8942-4c7b-9223-89297ae6b9c9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:58.188067 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef222d53-8942-4c7b-9223-89297ae6b9c9 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:58.218855 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 822/1664] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:55:58 2022] GET /identity/v3/roles => generated 1420 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:55:58.223120 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0078db31-da11-43c8-b41b-777b0d5b131d admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:58.242198 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0078db31-da11-43c8-b41b-777b0d5b131d admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:58.262839 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ad06499-8b02-4a2d-a3e5-a7af8df7a720 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:58.262839 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad06499-8b02-4a2d-a3e5-a7af8df7a720 admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:58.262839 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad06499-8b02-4a2d-a3e5-a7af8df7a720 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:58.262839 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad06499-8b02-4a2d-a3e5-a7af8df7a720 admin admin] PATH_INFO: `/v3/projects/54e1d6edeaa7407db1d7c18065755428/users/b476c50fa4634227a82abeec222c137d/roles/092036ee46ac427baf4d56468575f697` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:58.293180 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ad06499-8b02-4a2d-a3e5-a7af8df7a720 admin admin] RBAC: Authorizing `identity:create_grant(project_id=54e1d6edeaa7407db1d7c18065755428, user_id=b476c50fa4634227a82abeec222c137d, role_id=092036ee46ac427baf4d56468575f697)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:58.296538 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ad06499-8b02-4a2d-a3e5-a7af8df7a720 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:58.326856 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 843/1665] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:55:58 2022] PUT /identity/v3/projects/54e1d6edeaa7407db1d7c18065755428/users/b476c50fa4634227a82abeec222c137d/roles/092036ee46ac427baf4d56468575f697 => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:55:58.330569 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef222d53-8942-4c7b-9223-89297ae6b9c9 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:58.344810 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef222d53-8942-4c7b-9223-89297ae6b9c9 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:58.504316 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6bd7c5f-3ce9-4379-a6f9-cb0d006aab32 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:58.505058 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6bd7c5f-3ce9-4379-a6f9-cb0d006aab32 admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:58.505283 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6bd7c5f-3ce9-4379-a6f9-cb0d006aab32 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:58.505558 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6bd7c5f-3ce9-4379-a6f9-cb0d006aab32 admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:58.508254 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6bd7c5f-3ce9-4379-a6f9-cb0d006aab32 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:58.509941 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6bd7c5f-3ce9-4379-a6f9-cb0d006aab32 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:58.532919 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 823/1666] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:55:58 2022] GET /identity/v3/roles => generated 1420 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 09 00:55:58.537070 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ad06499-8b02-4a2d-a3e5-a7af8df7a720 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:58.554005 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ad06499-8b02-4a2d-a3e5-a7af8df7a720 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:58.564640 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76e27d4e-6ebb-4bd1-90ce-a18e2377f30f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:58.565400 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e27d4e-6ebb-4bd1-90ce-a18e2377f30f admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:58.565853 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e27d4e-6ebb-4bd1-90ce-a18e2377f30f admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:58.566396 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e27d4e-6ebb-4bd1-90ce-a18e2377f30f admin admin] PATH_INFO: `/v3/projects/54e1d6edeaa7407db1d7c18065755428/users/b476c50fa4634227a82abeec222c137d/roles/092036ee46ac427baf4d56468575f697` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:58.569256 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76e27d4e-6ebb-4bd1-90ce-a18e2377f30f admin admin] RBAC: Authorizing `identity:create_grant(project_id=54e1d6edeaa7407db1d7c18065755428, user_id=b476c50fa4634227a82abeec222c137d, role_id=092036ee46ac427baf4d56468575f697)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:58.576943 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76e27d4e-6ebb-4bd1-90ce-a18e2377f30f admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:58.597709 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 844/1667] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:55:58 2022] PUT /identity/v3/projects/54e1d6edeaa7407db1d7c18065755428/users/b476c50fa4634227a82abeec222c137d/roles/092036ee46ac427baf4d56468575f697 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:55:58.599102 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-903dc6e3-f339-4a29-9498-50c439ea99a6 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:58.599384 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-903dc6e3-f339-4a29-9498-50c439ea99a6 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:58.599699 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-903dc6e3-f339-4a29-9498-50c439ea99a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:58.676578 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-903dc6e3-f339-4a29-9498-50c439ea99a6 None None] MFA Rules not processed for user `b476c50fa4634227a82abeec222c137d`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:55:58.677812 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-903dc6e3-f339-4a29-9498-50c439ea99a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:55:58.759688 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:55:58.851725 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 824/1668] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:55:58 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 253 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:55:58.863378 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76e27d4e-6ebb-4bd1-90ce-a18e2377f30f admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:55:58.865495 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-76e27d4e-6ebb-4bd1-90ce-a18e2377f30f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:55:58.999491 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76e27d4e-6ebb-4bd1-90ce-a18e2377f30f admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:55:59.009547 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4720bf1b-28a7-4acc-9fa8-296d2f63b455 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3f48335668a4e8f934ac11ff66cd308', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:55:59.010189 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4720bf1b-28a7-4acc-9fa8-296d2f63b455 service cinder] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:59.010470 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4720bf1b-28a7-4acc-9fa8-296d2f63b455 service cinder] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:59.010746 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4720bf1b-28a7-4acc-9fa8-296d2f63b455 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:59.057738 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4720bf1b-28a7-4acc-9fa8-296d2f63b455 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:55:59.059215 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4720bf1b-28a7-4acc-9fa8-296d2f63b455 service cinder] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:55:59.070601 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 845/1669] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:55:58 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 208 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:55:59.791063 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875c5f3e-6c6a-469f-bebe-2fd49639174a None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:55:59.791362 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875c5f3e-6c6a-469f-bebe-2fd49639174a None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:55:59.791727 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875c5f3e-6c6a-469f-bebe-2fd49639174a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:55:59.838081 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-875c5f3e-6c6a-469f-bebe-2fd49639174a None None] MFA Rules not processed for user `1512c91201f14b75a9be35b89c2097ef`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:55:59.839637 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-875c5f3e-6c6a-469f-bebe-2fd49639174a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:55:59.920025 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 825/1670] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:55:59 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:56:00.540796 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c17df3-4041-4f02-aa81-b110dd20a360 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:00.541310 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c17df3-4041-4f02-aa81-b110dd20a360 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:00.541310 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c17df3-4041-4f02-aa81-b110dd20a360 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:00.615549 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-58c17df3-4041-4f02-aa81-b110dd20a360 None None] MFA Rules not processed for user `15e90d2d2b184f339d1d2aacb77bc82b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:56:00.615549 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-58c17df3-4041-4f02-aa81-b110dd20a360 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:00.762939 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:56:00.882363 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 846/1671] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:56:00 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 342 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:56:00.893161 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-964f2807-b37e-4045-a814-f991dcb6dc9f None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:00.893569 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-964f2807-b37e-4045-a814-f991dcb6dc9f None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:00.894084 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-964f2807-b37e-4045-a814-f991dcb6dc9f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:00.920310 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58c17df3-4041-4f02-aa81-b110dd20a360 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:00.921411 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-58c17df3-4041-4f02-aa81-b110dd20a360 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:00.955763 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-964f2807-b37e-4045-a814-f991dcb6dc9f None None] MFA Rules not processed for user `15e90d2d2b184f339d1d2aacb77bc82b`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:56:00.957275 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-964f2807-b37e-4045-a814-f991dcb6dc9f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:00.973729 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 826/1672] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:56:00 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:56:00.995519 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ebe111-9df0-416d-9ed7-0fcaf20e40f7 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:00.995835 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ebe111-9df0-416d-9ed7-0fcaf20e40f7 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:00.996116 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ebe111-9df0-416d-9ed7-0fcaf20e40f7 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:00.997492 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 827/1673] 10.222.0.33 () {54 vars in 912 bytes} [Sun Oct 9 00:56:00 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:56:01.003124 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1ebe111-9df0-416d-9ed7-0fcaf20e40f7 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:01.010215 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-f1ebe111-9df0-416d-9ed7-0fcaf20e40f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:01.053610 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58c17df3-4041-4f02-aa81-b110dd20a360 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:01.059171 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7332a99-3898-4b51-a77e-c3558fde9efb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:01.060670 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7332a99-3898-4b51-a77e-c3558fde9efb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:01.060957 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7332a99-3898-4b51-a77e-c3558fde9efb admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:01.061231 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7332a99-3898-4b51-a77e-c3558fde9efb admin admin] PATH_INFO: `/v3/users/8acc3f6ca4a84de99de98217f7d36d81` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:01.064590 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7332a99-3898-4b51-a77e-c3558fde9efb admin admin] RBAC: Authorizing `identity:delete_user(user_id=8acc3f6ca4a84de99de98217f7d36d81)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:01.067931 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7332a99-3898-4b51-a77e-c3558fde9efb admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:01.162087 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1ebe111-9df0-416d-9ed7-0fcaf20e40f7 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:01.181607 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a256a42c-39ad-41c7-9f1e-bf498e1069f4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a5d95969e74157bb2d2a57207e1651', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:01.182259 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a256a42c-39ad-41c7-9f1e-bf498e1069f4 service nova] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:01.182576 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a256a42c-39ad-41c7-9f1e-bf498e1069f4 service nova] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:01.183859 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a256a42c-39ad-41c7-9f1e-bf498e1069f4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:01.209616 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a256a42c-39ad-41c7-9f1e-bf498e1069f4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:01.211252 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a256a42c-39ad-41c7-9f1e-bf498e1069f4 service nova] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:01.221766 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-a7332a99-3898-4b51-a77e-c3558fde9efb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8acc3f6ca4a84de99de98217f7d36d81'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:01.232086 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-a7332a99-3898-4b51-a77e-c3558fde9efb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8acc3f6ca4a84de99de98217f7d36d81'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:01.232989 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-a7332a99-3898-4b51-a77e-c3558fde9efb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8acc3f6ca4a84de99de98217f7d36d81'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:01.234185 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-a7332a99-3898-4b51-a77e-c3558fde9efb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8acc3f6ca4a84de99de98217f7d36d81'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:01.257038 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 828/1674] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:56:00 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 259 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:56:01.259682 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-a7332a99-3898-4b51-a77e-c3558fde9efb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8acc3f6ca4a84de99de98217f7d36d81'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:01.261826 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd2440ba-e476-49c7-9b13-0318bceafeb7 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:01.262115 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd2440ba-e476-49c7-9b13-0318bceafeb7 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:01.262395 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd2440ba-e476-49c7-9b13-0318bceafeb7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:01.301716 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 847/1675] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 00:56:00 2022] DELETE /identity/v3/users/8acc3f6ca4a84de99de98217f7d36d81 => generated 0 bytes in 379 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:56:01.308378 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ea5ab12-065c-46a6-b697-976be819709d None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:01.308672 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ea5ab12-065c-46a6-b697-976be819709d None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:01.311100 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ea5ab12-065c-46a6-b697-976be819709d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:01.362940 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-1ea5ab12-065c-46a6-b697-976be819709d None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:56:01.365818 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-1ea5ab12-065c-46a6-b697-976be819709d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:01.389499 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-bd2440ba-e476-49c7-9b13-0318bceafeb7 None None] MFA Rules not processed for user `7d2eabf9865847ceb6e44d55126078a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:56:01.390676 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-bd2440ba-e476-49c7-9b13-0318bceafeb7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:01.510047 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:56:01.553922 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:56:01.588281 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 829/1676] 10.222.0.33 () {58 vars in 1009 bytes} [Sun Oct 9 00:56:01 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 331 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:56:01.596037 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 848/1677] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:56:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 293 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:56:01.678419 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd2440ba-e476-49c7-9b13-0318bceafeb7 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:01.679356 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ea5ab12-065c-46a6-b697-976be819709d None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:01.680134 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-bd2440ba-e476-49c7-9b13-0318bceafeb7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:01.680797 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-1ea5ab12-065c-46a6-b697-976be819709d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:01.702151 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ea5ab12-065c-46a6-b697-976be819709d None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:01.714583 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f595eb92-967d-4dd4-9579-6e6aa802b9bf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:01.715107 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd2440ba-e476-49c7-9b13-0318bceafeb7 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:01.715294 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f595eb92-967d-4dd4-9579-6e6aa802b9bf service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:01.715899 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f595eb92-967d-4dd4-9579-6e6aa802b9bf service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:01.715899 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f595eb92-967d-4dd4-9579-6e6aa802b9bf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:01.734958 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54724f35-d8c1-4332-8537-d9c008de1a5b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:01.738120 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54724f35-d8c1-4332-8537-d9c008de1a5b service neutron] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:01.738659 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f595eb92-967d-4dd4-9579-6e6aa802b9bf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:01.740681 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f595eb92-967d-4dd4-9579-6e6aa802b9bf service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:01.742146 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54724f35-d8c1-4332-8537-d9c008de1a5b service neutron] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:01.742883 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54724f35-d8c1-4332-8537-d9c008de1a5b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:01.771705 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 849/1678] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:56:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:56:01.785575 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54724f35-d8c1-4332-8537-d9c008de1a5b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:01.792082 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54724f35-d8c1-4332-8537-d9c008de1a5b service neutron] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:01.818984 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 830/1679] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:56:01 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 141 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:56:02.510291 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8301b9f7-7db2-4d01-b6cb-8747a65ca0ec None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:02.510578 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8301b9f7-7db2-4d01-b6cb-8747a65ca0ec None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:02.510859 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8301b9f7-7db2-4d01-b6cb-8747a65ca0ec None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:02.513086 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 850/1680] 10.222.0.33 () {54 vars in 912 bytes} [Sun Oct 9 00:56:02 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:56:02.524917 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-156497aa-5d23-4db1-9e0c-a94b14928a77 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:02.525484 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-156497aa-5d23-4db1-9e0c-a94b14928a77 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:02.525484 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-156497aa-5d23-4db1-9e0c-a94b14928a77 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:02.577942 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-156497aa-5d23-4db1-9e0c-a94b14928a77 None None] MFA Rules not processed for user `7d2eabf9865847ceb6e44d55126078a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:56:02.581399 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-156497aa-5d23-4db1-9e0c-a94b14928a77 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:02.608977 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 831/1681] 10.222.0.33 () {58 vars in 1009 bytes} [Sun Oct 9 00:56:02 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:56:02.634682 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8301b9f7-7db2-4d01-b6cb-8747a65ca0ec None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:02.640485 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-8301b9f7-7db2-4d01-b6cb-8747a65ca0ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:02.681510 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8301b9f7-7db2-4d01-b6cb-8747a65ca0ec None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:02.698123 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-610f2381-05e9-4bed-a099-7d7ec332443d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:02.699213 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-610f2381-05e9-4bed-a099-7d7ec332443d service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:02.699714 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-610f2381-05e9-4bed-a099-7d7ec332443d service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:02.700127 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-610f2381-05e9-4bed-a099-7d7ec332443d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:02.719901 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-610f2381-05e9-4bed-a099-7d7ec332443d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:02.724049 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-610f2381-05e9-4bed-a099-7d7ec332443d service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:02.745378 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 851/1682] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:56:02 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:56:03.034383 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-156497aa-5d23-4db1-9e0c-a94b14928a77 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:03.035900 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-156497aa-5d23-4db1-9e0c-a94b14928a77 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:03.077003 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-156497aa-5d23-4db1-9e0c-a94b14928a77 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:03.106189 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-453d37e5-776c-4a8a-861a-7564c2fe85af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:03.107132 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453d37e5-776c-4a8a-861a-7564c2fe85af admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:03.107604 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453d37e5-776c-4a8a-861a-7564c2fe85af admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:03.109948 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453d37e5-776c-4a8a-861a-7564c2fe85af admin admin] PATH_INFO: `/v3/users/aaf45940259542d7bea290b45ac9db64` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:03.113258 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-453d37e5-776c-4a8a-861a-7564c2fe85af admin admin] RBAC: Authorizing `identity:delete_user(user_id=aaf45940259542d7bea290b45ac9db64)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:03.115357 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-453d37e5-776c-4a8a-861a-7564c2fe85af admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:03.224425 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-610f2381-05e9-4bed-a099-7d7ec332443d service neutron] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:03.226264 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-610f2381-05e9-4bed-a099-7d7ec332443d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:03.258751 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-453d37e5-776c-4a8a-861a-7564c2fe85af admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'aaf45940259542d7bea290b45ac9db64'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:03.270335 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-453d37e5-776c-4a8a-861a-7564c2fe85af admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'aaf45940259542d7bea290b45ac9db64'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:03.271294 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-453d37e5-776c-4a8a-861a-7564c2fe85af admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'aaf45940259542d7bea290b45ac9db64'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:03.285202 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-453d37e5-776c-4a8a-861a-7564c2fe85af admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'aaf45940259542d7bea290b45ac9db64'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:03.310535 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-453d37e5-776c-4a8a-861a-7564c2fe85af admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'aaf45940259542d7bea290b45ac9db64'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:03.322314 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 832/1683] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 00:56:03 2022] DELETE /identity/v3/users/aaf45940259542d7bea290b45ac9db64 => generated 0 bytes in 288 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:56:03.326397 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d492c92-5cf7-45f6-b0c9-0b29bf4b288c None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:03.326774 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d492c92-5cf7-45f6-b0c9-0b29bf4b288c None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:03.327379 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d492c92-5cf7-45f6-b0c9-0b29bf4b288c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:03.354110 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-3d492c92-5cf7-45f6-b0c9-0b29bf4b288c None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:56:03.355425 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-3d492c92-5cf7-45f6-b0c9-0b29bf4b288c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:03.387282 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-610f2381-05e9-4bed-a099-7d7ec332443d service neutron] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:03.444904 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 833/1684] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:56:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 09 00:56:03.444904 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d492c92-5cf7-45f6-b0c9-0b29bf4b288c None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:03.447347 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-3d492c92-5cf7-45f6-b0c9-0b29bf4b288c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:03.475742 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d492c92-5cf7-45f6-b0c9-0b29bf4b288c None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:03.492265 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6b4c848-5e15-4f84-add0-e455f53546e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:03.492919 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6b4c848-5e15-4f84-add0-e455f53546e0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:03.493200 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6b4c848-5e15-4f84-add0-e455f53546e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:03.493473 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6b4c848-5e15-4f84-add0-e455f53546e0 admin admin] PATH_INFO: `/v3/projects/197f7f9effc24c8eb190027e4d47cda3` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:03.494709 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6b4c848-5e15-4f84-add0-e455f53546e0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=197f7f9effc24c8eb190027e4d47cda3)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:03.496703 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6b4c848-5e15-4f84-add0-e455f53546e0 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:03.510994 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de7ad515-d25c-4d41-ab89-f3bc51cd1135 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd25e240cb93643ed8122ccac9586936f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:03.512357 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de7ad515-d25c-4d41-ab89-f3bc51cd1135 service placement] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:03.512749 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de7ad515-d25c-4d41-ab89-f3bc51cd1135 service placement] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:03.513801 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de7ad515-d25c-4d41-ab89-f3bc51cd1135 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:03.516785 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-de7ad515-d25c-4d41-ab89-f3bc51cd1135 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:03.539312 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de7ad515-d25c-4d41-ab89-f3bc51cd1135 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:03.541810 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de7ad515-d25c-4d41-ab89-f3bc51cd1135 service placement] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:03.584521 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:56:03.585021 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 852/1685] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:56:03 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 361 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:56:03.587749 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de7ad515-d25c-4d41-ab89-f3bc51cd1135 service placement] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:03.593250 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-de7ad515-d25c-4d41-ab89-f3bc51cd1135 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:03.613023 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-c6b4c848-5e15-4f84-add0-e455f53546e0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '197f7f9effc24c8eb190027e4d47cda3', 'request_id': 'req-c6b4c848-5e15-4f84-add0-e455f53546e0'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:03.621955 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-c6b4c848-5e15-4f84-add0-e455f53546e0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '197f7f9effc24c8eb190027e4d47cda3', 'request_id': 'req-c6b4c848-5e15-4f84-add0-e455f53546e0'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:03.632216 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-c6b4c848-5e15-4f84-add0-e455f53546e0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '197f7f9effc24c8eb190027e4d47cda3', 'request_id': 'req-c6b4c848-5e15-4f84-add0-e455f53546e0'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:03.663957 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-c6b4c848-5e15-4f84-add0-e455f53546e0 admin admin] The token cache is being invalidate because project 197f7f9effc24c8eb190027e4d47cda3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81453) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 00:56:03.664483 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-c6b4c848-5e15-4f84-add0-e455f53546e0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:03.665284 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-c6b4c848-5e15-4f84-add0-e455f53546e0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:03.668296 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 834/1686] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 00:56:03 2022] DELETE /identity/v3/projects/197f7f9effc24c8eb190027e4d47cda3 => generated 0 bytes in 225 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:56:03.675822 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa62a42b-3b19-49cb-903a-cd232bbbb0b8 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:03.680951 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa62a42b-3b19-49cb-903a-cd232bbbb0b8 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:03.681885 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa62a42b-3b19-49cb-903a-cd232bbbb0b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:03.687469 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de7ad515-d25c-4d41-ab89-f3bc51cd1135 service placement] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:03.715210 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:56:03.771454 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-fa62a42b-3b19-49cb-903a-cd232bbbb0b8 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:56:03.772009 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eb551ca-4f5f-4cc2-b505-1772fbea4d03 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:03.772879 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-fa62a42b-3b19-49cb-903a-cd232bbbb0b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:03.773427 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eb551ca-4f5f-4cc2-b505-1772fbea4d03 service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:03.774723 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eb551ca-4f5f-4cc2-b505-1772fbea4d03 service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:03.775213 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eb551ca-4f5f-4cc2-b505-1772fbea4d03 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:03.777520 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-4eb551ca-4f5f-4cc2-b505-1772fbea4d03 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:03.998115 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 835/1687] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:56:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 323 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:56:04.005295 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24bfb523-3bb5-4d3a-8416-8ecbdc76a00e None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:04.005688 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24bfb523-3bb5-4d3a-8416-8ecbdc76a00e None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:04.006061 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24bfb523-3bb5-4d3a-8416-8ecbdc76a00e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:04.070427 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-24bfb523-3bb5-4d3a-8416-8ecbdc76a00e None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:56:04.071928 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-24bfb523-3bb5-4d3a-8416-8ecbdc76a00e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:04.077022 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eb551ca-4f5f-4cc2-b505-1772fbea4d03 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:04.086577 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eb551ca-4f5f-4cc2-b505-1772fbea4d03 service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:04.102582 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 836/1688] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:56:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 09 00:56:04.103498 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24bfb523-3bb5-4d3a-8416-8ecbdc76a00e None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:04.108298 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-24bfb523-3bb5-4d3a-8416-8ecbdc76a00e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:04.147145 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 853/1689] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:56:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 562 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:56:04.147983 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24bfb523-3bb5-4d3a-8416-8ecbdc76a00e None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:04.151479 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eb551ca-4f5f-4cc2-b505-1772fbea4d03 service neutron] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:04.164987 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13180eca-0aa3-4c82-9d33-b73741fe8bb5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:04.165710 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13180eca-0aa3-4c82-9d33-b73741fe8bb5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:04.165906 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13180eca-0aa3-4c82-9d33-b73741fe8bb5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:04.166175 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13180eca-0aa3-4c82-9d33-b73741fe8bb5 admin admin] PATH_INFO: `/v3/users/87513018217c45a1be915892e301fc0b` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:04.173970 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13180eca-0aa3-4c82-9d33-b73741fe8bb5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=87513018217c45a1be915892e301fc0b)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:04.178934 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13180eca-0aa3-4c82-9d33-b73741fe8bb5 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:04.201950 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eb551ca-4f5f-4cc2-b505-1772fbea4d03 service neutron] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:04.221173 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-753f3a80-cb94-4012-b9c3-0514002a6e8a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:04.223017 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-753f3a80-cb94-4012-b9c3-0514002a6e8a admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:04.224256 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-753f3a80-cb94-4012-b9c3-0514002a6e8a admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:04.224817 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-753f3a80-cb94-4012-b9c3-0514002a6e8a admin admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:04.234509 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-753f3a80-cb94-4012-b9c3-0514002a6e8a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:56:04.235040 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-753f3a80-cb94-4012-b9c3-0514002a6e8a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:04.238758 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-753f3a80-cb94-4012-b9c3-0514002a6e8a admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:04.263664 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 854/1690] 10.222.0.33 () {58 vars in 1153 bytes} [Sun Oct 9 00:56:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 116 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:56:04.272201 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-753f3a80-cb94-4012-b9c3-0514002a6e8a admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:04.322363 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-753f3a80-cb94-4012-b9c3-0514002a6e8a admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:04.335342 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-13180eca-0aa3-4c82-9d33-b73741fe8bb5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '87513018217c45a1be915892e301fc0b'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:04.340369 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7699143d-b6af-4b2a-b179-b41ccf2aaaed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:04.343366 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7699143d-b6af-4b2a-b179-b41ccf2aaaed admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:04.347907 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7699143d-b6af-4b2a-b179-b41ccf2aaaed admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:04.350511 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7699143d-b6af-4b2a-b179-b41ccf2aaaed admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:04.351917 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7699143d-b6af-4b2a-b179-b41ccf2aaaed admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:04.358750 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7699143d-b6af-4b2a-b179-b41ccf2aaaed admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:04.363171 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-13180eca-0aa3-4c82-9d33-b73741fe8bb5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '87513018217c45a1be915892e301fc0b'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:04.364159 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-13180eca-0aa3-4c82-9d33-b73741fe8bb5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '87513018217c45a1be915892e301fc0b'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:04.385729 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-13180eca-0aa3-4c82-9d33-b73741fe8bb5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '87513018217c45a1be915892e301fc0b'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:04.407236 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-13180eca-0aa3-4c82-9d33-b73741fe8bb5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '87513018217c45a1be915892e301fc0b'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:04.428431 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 855/1691] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 00:56:04 2022] POST /identity/v3/projects => generated 403 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:56:04.428814 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 837/1692] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 00:56:04 2022] DELETE /identity/v3/users/87513018217c45a1be915892e301fc0b => generated 0 bytes in 326 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:56:04.434600 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7699143d-b6af-4b2a-b179-b41ccf2aaaed admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:04.438606 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c191273-c1e3-4744-a264-c285ef6e7622 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:04.439104 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c191273-c1e3-4744-a264-c285ef6e7622 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:04.440302 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c191273-c1e3-4744-a264-c285ef6e7622 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:04.440723 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-7699143d-b6af-4b2a-b179-b41ccf2aaaed admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:04.510862 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-1c191273-c1e3-4744-a264-c285ef6e7622 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:56:04.521571 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-1c191273-c1e3-4744-a264-c285ef6e7622 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:04.591304 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:DELETE=15,keystone:SELECT=83,keystone:INSERT=2 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:56:04.668668 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7699143d-b6af-4b2a-b179-b41ccf2aaaed admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:04.705715 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 838/1693] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:56:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 270 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:56:04.760026 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c191273-c1e3-4744-a264-c285ef6e7622 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:04.762178 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-1c191273-c1e3-4744-a264-c285ef6e7622 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:04.798491 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:56:04.841391 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13f6fec7-d76c-42bd-a2d0-c7412194342a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:04.842160 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f6fec7-d76c-42bd-a2d0-c7412194342a admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:04.842443 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f6fec7-d76c-42bd-a2d0-c7412194342a admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:04.842716 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f6fec7-d76c-42bd-a2d0-c7412194342a admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:04.843405 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13f6fec7-d76c-42bd-a2d0-c7412194342a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:04.846384 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13f6fec7-d76c-42bd-a2d0-c7412194342a admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:04.904715 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 856/1694] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:56:04 2022] POST /identity/v3/users => generated 302 bytes in 471 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:56:04.907037 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13f6fec7-d76c-42bd-a2d0-c7412194342a admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:04.929332 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13f6fec7-d76c-42bd-a2d0-c7412194342a admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:04.944333 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa300f5b-25d0-4376-b6c5-0294c00b3de7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:04.944966 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa300f5b-25d0-4376-b6c5-0294c00b3de7 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:04.949026 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa300f5b-25d0-4376-b6c5-0294c00b3de7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:04.949423 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa300f5b-25d0-4376-b6c5-0294c00b3de7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:04.950537 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c191273-c1e3-4744-a264-c285ef6e7622 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:04.951319 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa300f5b-25d0-4376-b6c5-0294c00b3de7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:04.955823 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa300f5b-25d0-4376-b6c5-0294c00b3de7 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:04.973965 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7de8fee5-92f4-46b2-a6f9-04bf10dceffe service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:04.975304 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7de8fee5-92f4-46b2-a6f9-04bf10dceffe service neutron] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:04.975765 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7de8fee5-92f4-46b2-a6f9-04bf10dceffe service neutron] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:04.976200 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7de8fee5-92f4-46b2-a6f9-04bf10dceffe service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:05.007239 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7de8fee5-92f4-46b2-a6f9-04bf10dceffe service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:05.008907 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 857/1695] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:56:04 2022] GET /identity/v3/roles => generated 1420 bytes in 104 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:56:05.012759 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7de8fee5-92f4-46b2-a6f9-04bf10dceffe service neutron] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:05.012956 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa300f5b-25d0-4376-b6c5-0294c00b3de7 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:05.042667 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 839/1696] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:56:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 284 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 09 00:56:05.074991 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa300f5b-25d0-4376-b6c5-0294c00b3de7 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:05.100445 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05eb59c3-3260-4586-b9b7-8d0134dea5f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:05.101079 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05eb59c3-3260-4586-b9b7-8d0134dea5f5 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:05.101358 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05eb59c3-3260-4586-b9b7-8d0134dea5f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:05.101628 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05eb59c3-3260-4586-b9b7-8d0134dea5f5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:05.102309 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05eb59c3-3260-4586-b9b7-8d0134dea5f5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:05.107000 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05eb59c3-3260-4586-b9b7-8d0134dea5f5 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:05.144552 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 858/1697] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:56:05 2022] GET /identity/v3/roles => generated 1420 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:56:05.149244 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7de8fee5-92f4-46b2-a6f9-04bf10dceffe service neutron] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:05.169223 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7de8fee5-92f4-46b2-a6f9-04bf10dceffe service neutron] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:05.183222 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22b7d4e4-d923-47ca-beb3-176f90245122 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:05.183995 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b7d4e4-d923-47ca-beb3-176f90245122 admin admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:05.184277 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b7d4e4-d923-47ca-beb3-176f90245122 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:05.184558 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b7d4e4-d923-47ca-beb3-176f90245122 admin admin] PATH_INFO: `/v3/projects/46dad7bb16e340d9b0e65ac40a615b91/users/b0ed8dd01c2848839e45b6651a32d3e5/roles/588016933bab436d8b7c3ac76827356d` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:05.257921 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22b7d4e4-d923-47ca-beb3-176f90245122 admin admin] RBAC: Authorizing `identity:create_grant(project_id=46dad7bb16e340d9b0e65ac40a615b91, user_id=b0ed8dd01c2848839e45b6651a32d3e5, role_id=588016933bab436d8b7c3ac76827356d)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:05.279775 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22b7d4e4-d923-47ca-beb3-176f90245122 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:05.353270 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 840/1698] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:56:05 2022] PUT /identity/v3/projects/46dad7bb16e340d9b0e65ac40a615b91/users/b0ed8dd01c2848839e45b6651a32d3e5/roles/588016933bab436d8b7c3ac76827356d => generated 0 bytes in 204 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:56:05.368414 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6a5c66-aeee-4e24-b1b3-4ed89e8dffad None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:05.368704 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6a5c66-aeee-4e24-b1b3-4ed89e8dffad None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:05.368976 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6a5c66-aeee-4e24-b1b3-4ed89e8dffad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:05.506278 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-2e6a5c66-aeee-4e24-b1b3-4ed89e8dffad None None] MFA Rules not processed for user `b0ed8dd01c2848839e45b6651a32d3e5`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:56:05.512511 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-2e6a5c66-aeee-4e24-b1b3-4ed89e8dffad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:05.576262 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22b7d4e4-d923-47ca-beb3-176f90245122 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:05.579946 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-22b7d4e4-d923-47ca-beb3-176f90245122 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:05.781950 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 859/1699] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:56:05 2022] POST /identity/v3/auth/tokens => generated 3280 bytes in 414 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:56:05.782694 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e6a5c66-aeee-4e24-b1b3-4ed89e8dffad None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:05.826506 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22b7d4e4-d923-47ca-beb3-176f90245122 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:05.841177 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b02c6a23-09a0-4682-b3e9-4bd159714dfe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:05.841458 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e6a5c66-aeee-4e24-b1b3-4ed89e8dffad None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:05.844558 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b02c6a23-09a0-4682-b3e9-4bd159714dfe admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:05.846405 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b02c6a23-09a0-4682-b3e9-4bd159714dfe admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:05.846405 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b02c6a23-09a0-4682-b3e9-4bd159714dfe admin admin] PATH_INFO: `/v3/projects/c283e71634c340be94a2248f9a5f9b82` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:05.848676 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b02c6a23-09a0-4682-b3e9-4bd159714dfe admin admin] RBAC: Authorizing `identity:delete_project(project_id=c283e71634c340be94a2248f9a5f9b82)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:05.867922 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9ad5bd9-07d3-4811-a61a-3b300ab20839 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:05.867922 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ad5bd9-07d3-4811-a61a-3b300ab20839 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:05.868172 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ad5bd9-07d3-4811-a61a-3b300ab20839 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:05.869315 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b02c6a23-09a0-4682-b3e9-4bd159714dfe admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:05.874764 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ad5bd9-07d3-4811-a61a-3b300ab20839 admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:05.875544 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9ad5bd9-07d3-4811-a61a-3b300ab20839 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:05.881046 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9ad5bd9-07d3-4811-a61a-3b300ab20839 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:06.012680 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 860/1700] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 00:56:05 2022] POST /identity/v3/projects => generated 405 bytes in 230 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:56:06.016519 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9ad5bd9-07d3-4811-a61a-3b300ab20839 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:06.025668 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-b02c6a23-09a0-4682-b3e9-4bd159714dfe admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c283e71634c340be94a2248f9a5f9b82', 'request_id': 'req-b02c6a23-09a0-4682-b3e9-4bd159714dfe'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:06.027894 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=95,keystone:INSERT=1,keystone:DELETE=4 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:56:06.039141 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-b02c6a23-09a0-4682-b3e9-4bd159714dfe admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c283e71634c340be94a2248f9a5f9b82', 'request_id': 'req-b02c6a23-09a0-4682-b3e9-4bd159714dfe'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:06.054194 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-b02c6a23-09a0-4682-b3e9-4bd159714dfe admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c283e71634c340be94a2248f9a5f9b82', 'request_id': 'req-b02c6a23-09a0-4682-b3e9-4bd159714dfe'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:06.093415 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-b02c6a23-09a0-4682-b3e9-4bd159714dfe admin admin] The token cache is being invalidate because project c283e71634c340be94a2248f9a5f9b82 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81453) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 00:56:06.093714 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-b02c6a23-09a0-4682-b3e9-4bd159714dfe admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:06.096183 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-b02c6a23-09a0-4682-b3e9-4bd159714dfe admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:06.100224 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 841/1701] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 00:56:05 2022] DELETE /identity/v3/projects/c283e71634c340be94a2248f9a5f9b82 => generated 0 bytes in 525 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:56:06.105872 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b02c6a23-09a0-4682-b3e9-4bd159714dfe admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:06.107632 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-b02c6a23-09a0-4682-b3e9-4bd159714dfe admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:06.246454 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9ad5bd9-07d3-4811-a61a-3b300ab20839 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:06.271317 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-715d9eb6-dea0-450f-9780-afedd33c051f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:06.272037 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-715d9eb6-dea0-450f-9780-afedd33c051f admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:06.273269 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-715d9eb6-dea0-450f-9780-afedd33c051f admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:06.273695 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-715d9eb6-dea0-450f-9780-afedd33c051f admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:06.277902 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-715d9eb6-dea0-450f-9780-afedd33c051f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:06.280421 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-715d9eb6-dea0-450f-9780-afedd33c051f admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:06.327456 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b02c6a23-09a0-4682-b3e9-4bd159714dfe admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:06.342394 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 861/1702] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:56:06 2022] POST /identity/v3/users => generated 309 bytes in 326 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 09 00:56:06.348082 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-715d9eb6-dea0-450f-9780-afedd33c051f admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:06.356003 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf9332a6-52a1-4bc6-8161-f31feb35fcd0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:06.362496 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf9332a6-52a1-4bc6-8161-f31feb35fcd0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:06.365497 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf9332a6-52a1-4bc6-8161-f31feb35fcd0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:06.366588 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf9332a6-52a1-4bc6-8161-f31feb35fcd0 admin admin] PATH_INFO: `/v3/users/b476c50fa4634227a82abeec222c137d` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:06.370639 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf9332a6-52a1-4bc6-8161-f31feb35fcd0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b476c50fa4634227a82abeec222c137d)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:06.373982 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-715d9eb6-dea0-450f-9780-afedd33c051f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:06.375465 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf9332a6-52a1-4bc6-8161-f31feb35fcd0 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:06.410485 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-715d9eb6-dea0-450f-9780-afedd33c051f admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:06.436374 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c84c4f24-68ba-415f-b8d8-4b7873f31611 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:06.437010 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c84c4f24-68ba-415f-b8d8-4b7873f31611 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:06.437291 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c84c4f24-68ba-415f-b8d8-4b7873f31611 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:06.437566 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c84c4f24-68ba-415f-b8d8-4b7873f31611 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:06.438260 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c84c4f24-68ba-415f-b8d8-4b7873f31611 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:06.440023 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c84c4f24-68ba-415f-b8d8-4b7873f31611 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:06.441601 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:56:06.500794 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 862/1703] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:56:06 2022] GET /identity/v3/roles => generated 1420 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:56:06.502568 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c84c4f24-68ba-415f-b8d8-4b7873f31611 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:06.536571 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c84c4f24-68ba-415f-b8d8-4b7873f31611 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:06.542373 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-bf9332a6-52a1-4bc6-8161-f31feb35fcd0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b476c50fa4634227a82abeec222c137d'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:06.559404 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3249bb49-9d98-415f-b23f-8ccf83415655 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:06.560231 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3249bb49-9d98-415f-b23f-8ccf83415655 admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:06.560516 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3249bb49-9d98-415f-b23f-8ccf83415655 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:06.560787 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3249bb49-9d98-415f-b23f-8ccf83415655 admin admin] PATH_INFO: `/v3/projects/28e69a2efede4f4a834de16f5712ec2e/users/70fdef2a2e194a18b5b80696a911ab1f/roles/092036ee46ac427baf4d56468575f697` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:06.572815 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-bf9332a6-52a1-4bc6-8161-f31feb35fcd0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b476c50fa4634227a82abeec222c137d'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:06.576305 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-bf9332a6-52a1-4bc6-8161-f31feb35fcd0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b476c50fa4634227a82abeec222c137d'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:06.605839 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-bf9332a6-52a1-4bc6-8161-f31feb35fcd0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b476c50fa4634227a82abeec222c137d'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:06.619342 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3249bb49-9d98-415f-b23f-8ccf83415655 admin admin] RBAC: Authorizing `identity:create_grant(project_id=28e69a2efede4f4a834de16f5712ec2e, user_id=70fdef2a2e194a18b5b80696a911ab1f, role_id=092036ee46ac427baf4d56468575f697)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:06.621988 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3249bb49-9d98-415f-b23f-8ccf83415655 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:06.641683 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-bf9332a6-52a1-4bc6-8161-f31feb35fcd0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b476c50fa4634227a82abeec222c137d'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:06.662113 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 863/1704] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:56:06 2022] PUT /identity/v3/projects/28e69a2efede4f4a834de16f5712ec2e/users/70fdef2a2e194a18b5b80696a911ab1f/roles/092036ee46ac427baf4d56468575f697 => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:56:06.666073 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3249bb49-9d98-415f-b23f-8ccf83415655 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:06.668655 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-3249bb49-9d98-415f-b23f-8ccf83415655 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:06.670518 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 842/1705] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 00:56:06 2022] DELETE /identity/v3/users/b476c50fa4634227a82abeec222c137d => generated 0 bytes in 565 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:56:06.815172 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d26364f-c975-4e26-ac79-a7a33dd3087f None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:06.817185 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d26364f-c975-4e26-ac79-a7a33dd3087f None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:06.817633 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d26364f-c975-4e26-ac79-a7a33dd3087f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:06.821569 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3249bb49-9d98-415f-b23f-8ccf83415655 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:06.918427 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d98ed63a-5240-480b-aeb3-ce6aa4cb7e84 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:06.919370 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d98ed63a-5240-480b-aeb3-ce6aa4cb7e84 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:06.919769 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d98ed63a-5240-480b-aeb3-ce6aa4cb7e84 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:06.920171 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d98ed63a-5240-480b-aeb3-ce6aa4cb7e84 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:06.920790 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-8d26364f-c975-4e26-ac79-a7a33dd3087f None None] MFA Rules not processed for user `b027fb8adadc488e975c4541e7ee6626`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:56:06.921206 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d98ed63a-5240-480b-aeb3-ce6aa4cb7e84 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:06.922239 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-8d26364f-c975-4e26-ac79-a7a33dd3087f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:06.923869 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d98ed63a-5240-480b-aeb3-ce6aa4cb7e84 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:06.960706 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 864/1706] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:56:06 2022] GET /identity/v3/roles => generated 1420 bytes in 295 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:56:06.965549 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d98ed63a-5240-480b-aeb3-ce6aa4cb7e84 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:06.979458 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d98ed63a-5240-480b-aeb3-ce6aa4cb7e84 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:06.985197 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a44732c-13bb-41b1-966c-f80369a3eba1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:06.985965 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a44732c-13bb-41b1-966c-f80369a3eba1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:06.986247 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a44732c-13bb-41b1-966c-f80369a3eba1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:06.986521 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a44732c-13bb-41b1-966c-f80369a3eba1 admin admin] PATH_INFO: `/v3/projects/28e69a2efede4f4a834de16f5712ec2e/users/70fdef2a2e194a18b5b80696a911ab1f/roles/092036ee46ac427baf4d56468575f697` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:06.988716 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a44732c-13bb-41b1-966c-f80369a3eba1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=28e69a2efede4f4a834de16f5712ec2e, user_id=70fdef2a2e194a18b5b80696a911ab1f, role_id=092036ee46ac427baf4d56468575f697)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:06.991854 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a44732c-13bb-41b1-966c-f80369a3eba1 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:07.004433 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 865/1707] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:56:06 2022] PUT /identity/v3/projects/28e69a2efede4f4a834de16f5712ec2e/users/70fdef2a2e194a18b5b80696a911ab1f/roles/092036ee46ac427baf4d56468575f697 => generated 0 bytes in 39 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:56:07.015548 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da1111ac-b6f1-45c2-a883-ad5f35f407c0 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:07.015869 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da1111ac-b6f1-45c2-a883-ad5f35f407c0 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:07.016150 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da1111ac-b6f1-45c2-a883-ad5f35f407c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:07.050967 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-da1111ac-b6f1-45c2-a883-ad5f35f407c0 None None] MFA Rules not processed for user `70fdef2a2e194a18b5b80696a911ab1f`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:56:07.052242 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-da1111ac-b6f1-45c2-a883-ad5f35f407c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:07.097181 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:DELETE=13,keystone:SELECT=85,keystone:INSERT=2 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:56:07.148536 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 843/1708] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:56:06 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 334 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:56:07.161266 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:56:07.163758 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d26364f-c975-4e26-ac79-a7a33dd3087f None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:07.166034 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-8d26364f-c975-4e26-ac79-a7a33dd3087f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:07.204607 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 866/1709] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:56:07 2022] POST /identity/v3/auth/tokens => generated 3349 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:56:07.205850 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da1111ac-b6f1-45c2-a883-ad5f35f407c0 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:07.208504 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-da1111ac-b6f1-45c2-a883-ad5f35f407c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:07.342899 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da1111ac-b6f1-45c2-a883-ad5f35f407c0 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:07.350703 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-801b5830-908a-4fe5-8b42-41baf61f2ece service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3f48335668a4e8f934ac11ff66cd308', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:07.351608 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-801b5830-908a-4fe5-8b42-41baf61f2ece service cinder] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:07.352148 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-801b5830-908a-4fe5-8b42-41baf61f2ece service cinder] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:07.352615 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-801b5830-908a-4fe5-8b42-41baf61f2ece service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:07.371679 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-801b5830-908a-4fe5-8b42-41baf61f2ece service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:07.374443 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-801b5830-908a-4fe5-8b42-41baf61f2ece service cinder] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:07.398541 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 867/1710] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:56:07 2022] GET /identity/v3/auth/tokens => generated 3349 bytes in 193 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:56:07.399504 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-801b5830-908a-4fe5-8b42-41baf61f2ece service cinder] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:07.403516 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-801b5830-908a-4fe5-8b42-41baf61f2ece service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:07.403936 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d26364f-c975-4e26-ac79-a7a33dd3087f None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:07.625796 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-801b5830-908a-4fe5-8b42-41baf61f2ece service cinder] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:07.644999 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19e4feb4-e878-4f03-b166-d1aeb64bd512 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:07.653441 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19e4feb4-e878-4f03-b166-d1aeb64bd512 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:07.653931 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19e4feb4-e878-4f03-b166-d1aeb64bd512 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:07.655250 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19e4feb4-e878-4f03-b166-d1aeb64bd512 admin admin] PATH_INFO: `/v3/projects/071e8edf3cea479baab9f37f6dd378e3` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:07.665592 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19e4feb4-e878-4f03-b166-d1aeb64bd512 admin admin] RBAC: Authorizing `identity:delete_project(project_id=071e8edf3cea479baab9f37f6dd378e3)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:07.666777 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19e4feb4-e878-4f03-b166-d1aeb64bd512 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:07.673194 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a45dc90d-4bca-48ea-ad80-1a6634490449 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a5d95969e74157bb2d2a57207e1651', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:07.675661 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a45dc90d-4bca-48ea-ad80-1a6634490449 service nova] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:07.676669 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a45dc90d-4bca-48ea-ad80-1a6634490449 service nova] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:07.676669 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a45dc90d-4bca-48ea-ad80-1a6634490449 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:07.780812 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-19e4feb4-e878-4f03-b166-d1aeb64bd512 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '071e8edf3cea479baab9f37f6dd378e3', 'request_id': 'req-19e4feb4-e878-4f03-b166-d1aeb64bd512'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:07.792809 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-19e4feb4-e878-4f03-b166-d1aeb64bd512 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '071e8edf3cea479baab9f37f6dd378e3', 'request_id': 'req-19e4feb4-e878-4f03-b166-d1aeb64bd512'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:07.813541 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-19e4feb4-e878-4f03-b166-d1aeb64bd512 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '071e8edf3cea479baab9f37f6dd378e3', 'request_id': 'req-19e4feb4-e878-4f03-b166-d1aeb64bd512'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:07.849464 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-19e4feb4-e878-4f03-b166-d1aeb64bd512 admin admin] The token cache is being invalidate because project 071e8edf3cea479baab9f37f6dd378e3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 00:56:07.850608 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-19e4feb4-e878-4f03-b166-d1aeb64bd512 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:07.851789 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-19e4feb4-e878-4f03-b166-d1aeb64bd512 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:07.854609 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 868/1711] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 00:56:07 2022] DELETE /identity/v3/projects/071e8edf3cea479baab9f37f6dd378e3 => generated 0 bytes in 455 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:56:07.856613 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19e4feb4-e878-4f03-b166-d1aeb64bd512 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:07.859084 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-19e4feb4-e878-4f03-b166-d1aeb64bd512 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:07.929462 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a45dc90d-4bca-48ea-ad80-1a6634490449 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:07.931250 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:56:07.936880 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a45dc90d-4bca-48ea-ad80-1a6634490449 service nova] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:07.990413 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 844/1712] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:56:07 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 827 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 09 00:56:07.992035 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c402b5e3-dfe0-40ff-bec9-38015110e95a None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:07.992485 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c402b5e3-dfe0-40ff-bec9-38015110e95a None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:07.992909 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c402b5e3-dfe0-40ff-bec9-38015110e95a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:08.035523 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-c402b5e3-dfe0-40ff-bec9-38015110e95a None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:56:08.040521 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-c402b5e3-dfe0-40ff-bec9-38015110e95a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:08.081667 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:56:08.081873 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19e4feb4-e878-4f03-b166-d1aeb64bd512 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:08.098332 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d743730c-37ef-4202-a116-3623f3082bad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:08.099066 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d743730c-37ef-4202-a116-3623f3082bad admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:08.099257 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d743730c-37ef-4202-a116-3623f3082bad admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:08.099524 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d743730c-37ef-4202-a116-3623f3082bad admin admin] PATH_INFO: `/v3/users/b0ed8dd01c2848839e45b6651a32d3e5` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:08.102777 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d743730c-37ef-4202-a116-3623f3082bad admin admin] RBAC: Authorizing `identity:delete_user(user_id=b0ed8dd01c2848839e45b6651a32d3e5)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:08.106195 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d743730c-37ef-4202-a116-3623f3082bad admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:08.220952 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-d743730c-37ef-4202-a116-3623f3082bad admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b0ed8dd01c2848839e45b6651a32d3e5'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:08.243249 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 845/1713] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:56:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 252 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:56:08.251560 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-477fce64-b953-4102-ba2f-5057f18f23a8 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:08.252748 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-477fce64-b953-4102-ba2f-5057f18f23a8 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:08.253219 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-477fce64-b953-4102-ba2f-5057f18f23a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:08.254974 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-d743730c-37ef-4202-a116-3623f3082bad admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b0ed8dd01c2848839e45b6651a32d3e5'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:08.255892 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-d743730c-37ef-4202-a116-3623f3082bad admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b0ed8dd01c2848839e45b6651a32d3e5'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:08.257426 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-d743730c-37ef-4202-a116-3623f3082bad admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b0ed8dd01c2848839e45b6651a32d3e5'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:08.273994 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-d743730c-37ef-4202-a116-3623f3082bad admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b0ed8dd01c2848839e45b6651a32d3e5'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:08.292600 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-477fce64-b953-4102-ba2f-5057f18f23a8 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:56:08.294145 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-477fce64-b953-4102-ba2f-5057f18f23a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:08.299481 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 869/1714] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 00:56:07 2022] DELETE /identity/v3/users/b0ed8dd01c2848839e45b6651a32d3e5 => generated 0 bytes in 444 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:56:08.304785 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-222cfe93-4509-469b-b7e4-98b4459379e6 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:08.305171 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-222cfe93-4509-469b-b7e4-98b4459379e6 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:08.305514 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-222cfe93-4509-469b-b7e4-98b4459379e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:08.334409 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-222cfe93-4509-469b-b7e4-98b4459379e6 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:56:08.336032 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-222cfe93-4509-469b-b7e4-98b4459379e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:08.437954 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 846/1715] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:56:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:56:08.437954 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-477fce64-b953-4102-ba2f-5057f18f23a8 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:08.439304 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-477fce64-b953-4102-ba2f-5057f18f23a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:08.473031 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-477fce64-b953-4102-ba2f-5057f18f23a8 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:08.478096 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 870/1716] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:56:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 09 00:56:08.479583 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-222cfe93-4509-469b-b7e4-98b4459379e6 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:08.481615 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7b8d2fe-34b3-445a-a121-6939d14ed23d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:08.482340 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b8d2fe-34b3-445a-a121-6939d14ed23d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:08.482813 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b8d2fe-34b3-445a-a121-6939d14ed23d admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:08.483166 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b8d2fe-34b3-445a-a121-6939d14ed23d admin admin] PATH_INFO: `/v3/projects/54e1d6edeaa7407db1d7c18065755428` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:08.484459 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7b8d2fe-34b3-445a-a121-6939d14ed23d admin admin] RBAC: Authorizing `identity:delete_project(project_id=54e1d6edeaa7407db1d7c18065755428)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:08.487011 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7b8d2fe-34b3-445a-a121-6939d14ed23d admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:08.494919 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-222cfe93-4509-469b-b7e4-98b4459379e6 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:08.502449 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94c27006-0182-4fed-ba2d-49002131b832 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:08.503378 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94c27006-0182-4fed-ba2d-49002131b832 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:08.503885 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94c27006-0182-4fed-ba2d-49002131b832 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:08.504324 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94c27006-0182-4fed-ba2d-49002131b832 admin admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:08.505153 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94c27006-0182-4fed-ba2d-49002131b832 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:56:08.505670 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94c27006-0182-4fed-ba2d-49002131b832 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:08.507848 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94c27006-0182-4fed-ba2d-49002131b832 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:08.526964 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 871/1717] 10.222.0.33 () {58 vars in 1153 bytes} [Sun Oct 9 00:56:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:56:08.529610 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f63d98fe-c9e0-421d-b0c1-1deb1124b2b7 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:08.530290 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f63d98fe-c9e0-421d-b0c1-1deb1124b2b7 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:08.531138 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f63d98fe-c9e0-421d-b0c1-1deb1124b2b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:08.588548 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-a7b8d2fe-34b3-445a-a121-6939d14ed23d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '54e1d6edeaa7407db1d7c18065755428', 'request_id': 'req-a7b8d2fe-34b3-445a-a121-6939d14ed23d'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:08.608596 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-a7b8d2fe-34b3-445a-a121-6939d14ed23d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '54e1d6edeaa7407db1d7c18065755428', 'request_id': 'req-a7b8d2fe-34b3-445a-a121-6939d14ed23d'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:08.616546 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:56:08.618072 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-a7b8d2fe-34b3-445a-a121-6939d14ed23d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '54e1d6edeaa7407db1d7c18065755428', 'request_id': 'req-a7b8d2fe-34b3-445a-a121-6939d14ed23d'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:08.638699 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-f63d98fe-c9e0-421d-b0c1-1deb1124b2b7 None None] MFA Rules not processed for user `5f3247ddcda54394995d844c776bf43c`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:56:08.640621 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-f63d98fe-c9e0-421d-b0c1-1deb1124b2b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:08.651979 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-a7b8d2fe-34b3-445a-a121-6939d14ed23d admin admin] The token cache is being invalidate because project 54e1d6edeaa7407db1d7c18065755428 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81453) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 00:56:08.652419 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-a7b8d2fe-34b3-445a-a121-6939d14ed23d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:08.653610 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-a7b8d2fe-34b3-445a-a121-6939d14ed23d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:08.656572 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 847/1718] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 00:56:08 2022] DELETE /identity/v3/projects/54e1d6edeaa7407db1d7c18065755428 => generated 0 bytes in 221 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:56:08.658097 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7b8d2fe-34b3-445a-a121-6939d14ed23d admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:08.661480 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-a7b8d2fe-34b3-445a-a121-6939d14ed23d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:08.768101 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:56:08.799324 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7b8d2fe-34b3-445a-a121-6939d14ed23d admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:08.813168 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db9e7468-f81a-4af1-8667-09a04c988987 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:08.813856 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db9e7468-f81a-4af1-8667-09a04c988987 admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:08.814649 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db9e7468-f81a-4af1-8667-09a04c988987 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:08.814924 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db9e7468-f81a-4af1-8667-09a04c988987 admin admin] PATH_INFO: `/v3/projects` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:08.815568 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db9e7468-f81a-4af1-8667-09a04c988987 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:08.817889 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db9e7468-f81a-4af1-8667-09a04c988987 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:08.829200 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 872/1719] 10.222.0.33 () {58 vars in 1009 bytes} [Sun Oct 9 00:56:08 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 301 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:56:08.838418 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f63d98fe-c9e0-421d-b0c1-1deb1124b2b7 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:08.846835 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-f63d98fe-c9e0-421d-b0c1-1deb1124b2b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:08.881828 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 848/1720] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 00:56:08 2022] POST /identity/v3/projects => generated 365 bytes in 225 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:56:08.883122 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e11aa8e6-67db-41d4-a5c8-9470e66d40dd None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:08.883411 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e11aa8e6-67db-41d4-a5c8-9470e66d40dd None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:08.883713 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e11aa8e6-67db-41d4-a5c8-9470e66d40dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:08.932103 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-e11aa8e6-67db-41d4-a5c8-9470e66d40dd None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:56:08.933333 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-e11aa8e6-67db-41d4-a5c8-9470e66d40dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:08.993016 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f63d98fe-c9e0-421d-b0c1-1deb1124b2b7 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:09.066121 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 849/1721] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:56:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:56:09.067576 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d2e24af-9bd4-4a39-ae6d-e9f833840a0d None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:09.068055 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d2e24af-9bd4-4a39-ae6d-e9f833840a0d None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:09.068473 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d2e24af-9bd4-4a39-ae6d-e9f833840a0d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:09.138202 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d06fc60d-3675-4e4d-b83c-20cc4233d06b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:09.138912 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06fc60d-3675-4e4d-b83c-20cc4233d06b service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:09.139193 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06fc60d-3675-4e4d-b83c-20cc4233d06b service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:09.140261 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-4d2e24af-9bd4-4a39-ae6d-e9f833840a0d None None] MFA Rules not processed for user `438446d1e6b3402bb18b547c089391f8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:56:09.140460 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06fc60d-3675-4e4d-b83c-20cc4233d06b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:09.141942 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-4d2e24af-9bd4-4a39-ae6d-e9f833840a0d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:09.146006 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-d06fc60d-3675-4e4d-b83c-20cc4233d06b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:09.199411 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d06fc60d-3675-4e4d-b83c-20cc4233d06b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:09.205132 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d06fc60d-3675-4e4d-b83c-20cc4233d06b service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:09.235000 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 873/1722] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:56:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 405 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:56:09.236438 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d06fc60d-3675-4e4d-b83c-20cc4233d06b service neutron] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:09.281503 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d06fc60d-3675-4e4d-b83c-20cc4233d06b service neutron] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:09.292331 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 850/1723] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:56:09 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 225 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:56:09.302118 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-707a6e3e-2952-40eb-b7a0-b80cdd02089b None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:09.302118 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-707a6e3e-2952-40eb-b7a0-b80cdd02089b None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:09.302118 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-707a6e3e-2952-40eb-b7a0-b80cdd02089b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:09.302387 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d49303f-c488-4513-9a08-5a6c2fa36c86 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:09.303161 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d49303f-c488-4513-9a08-5a6c2fa36c86 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:09.303161 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d49303f-c488-4513-9a08-5a6c2fa36c86 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:09.303348 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d49303f-c488-4513-9a08-5a6c2fa36c86 admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:09.303981 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d49303f-c488-4513-9a08-5a6c2fa36c86 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:09.305922 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d49303f-c488-4513-9a08-5a6c2fa36c86 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:09.348823 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-707a6e3e-2952-40eb-b7a0-b80cdd02089b None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:56:09.350217 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-707a6e3e-2952-40eb-b7a0-b80cdd02089b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:09.369385 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 851/1724] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:56:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 76 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:56:09.370425 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-707a6e3e-2952-40eb-b7a0-b80cdd02089b None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:09.373111 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-707a6e3e-2952-40eb-b7a0-b80cdd02089b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:09.374548 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 874/1725] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:56:09 2022] POST /identity/v3/users => generated 283 bytes in 139 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:56:09.376211 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca5533a-25f0-4c93-997c-78f512365f42 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:09.377050 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca5533a-25f0-4c93-997c-78f512365f42 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:09.377328 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca5533a-25f0-4c93-997c-78f512365f42 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:09.443027 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-2ca5533a-25f0-4c93-997c-78f512365f42 None None] MFA Rules not processed for user `438446d1e6b3402bb18b547c089391f8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:56:09.449303 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-2ca5533a-25f0-4c93-997c-78f512365f42 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:09.472744 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 875/1726] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:56:09 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:56:09.474472 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ca5533a-25f0-4c93-997c-78f512365f42 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:09.507453 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:56:09.509537 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ca5533a-25f0-4c93-997c-78f512365f42 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:09.546916 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87460ddd-fcf6-4f0e-ae93-97de5434c6a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:09.549476 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87460ddd-fcf6-4f0e-ae93-97de5434c6a9 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:09.550151 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87460ddd-fcf6-4f0e-ae93-97de5434c6a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:09.550563 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87460ddd-fcf6-4f0e-ae93-97de5434c6a9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:09.551388 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87460ddd-fcf6-4f0e-ae93-97de5434c6a9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:09.556092 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87460ddd-fcf6-4f0e-ae93-97de5434c6a9 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:09.590526 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 876/1727] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:56:09 2022] GET /identity/v3/roles => generated 1420 bytes in 114 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:56:09.590526 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87460ddd-fcf6-4f0e-ae93-97de5434c6a9 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:09.591280 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-707a6e3e-2952-40eb-b7a0-b80cdd02089b None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:09.605174 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c98af4bb-89d0-49df-bf43-e903a198b1b6 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '70e367e802894b08afd7d756d8d9ecff', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:09.608444 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c98af4bb-89d0-49df-bf43-e903a198b1b6 service swift] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:09.609384 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c98af4bb-89d0-49df-bf43-e903a198b1b6 service swift] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:09.610639 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c98af4bb-89d0-49df-bf43-e903a198b1b6 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:09.612367 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87460ddd-fcf6-4f0e-ae93-97de5434c6a9 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:09.632418 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59c2f23e-f8de-4f3b-8f80-940121676e1f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:09.633053 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59c2f23e-f8de-4f3b-8f80-940121676e1f admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:09.633335 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59c2f23e-f8de-4f3b-8f80-940121676e1f admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:09.633610 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59c2f23e-f8de-4f3b-8f80-940121676e1f admin admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:09.634265 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59c2f23e-f8de-4f3b-8f80-940121676e1f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:56:09.634566 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59c2f23e-f8de-4f3b-8f80-940121676e1f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:09.636286 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59c2f23e-f8de-4f3b-8f80-940121676e1f admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:09.655423 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 877/1728] 10.222.0.33 () {58 vars in 1153 bytes} [Sun Oct 9 00:56:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 67 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:56:09.657001 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59c2f23e-f8de-4f3b-8f80-940121676e1f admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:09.663830 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-59c2f23e-f8de-4f3b-8f80-940121676e1f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:09.691161 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:56:09.748858 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c98af4bb-89d0-49df-bf43-e903a198b1b6 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:09.752793 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c98af4bb-89d0-49df-bf43-e903a198b1b6 service swift] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:09.793287 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59c2f23e-f8de-4f3b-8f80-940121676e1f admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:09.796117 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 852/1729] 10.222.0.33 () {60 vars in 1384 bytes} [Sun Oct 9 00:56:09 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 426 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 09 00:56:09.797569 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c98af4bb-89d0-49df-bf43-e903a198b1b6 service swift] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:09.810213 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a5a4e2-eb34-455d-862d-a5d495c7b8b0 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a5d95969e74157bb2d2a57207e1651', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:09.810213 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a5a4e2-eb34-455d-862d-a5d495c7b8b0 service nova] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:09.810213 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a5a4e2-eb34-455d-862d-a5d495c7b8b0 service nova] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:09.810213 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a5a4e2-eb34-455d-862d-a5d495c7b8b0 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:09.828737 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1a5a4e2-eb34-455d-862d-a5d495c7b8b0 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:09.829324 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c98af4bb-89d0-49df-bf43-e903a198b1b6 service swift] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:09.833175 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1a5a4e2-eb34-455d-862d-a5d495c7b8b0 service nova] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:09.835950 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd1d5fa0-2422-458d-a708-f1f4b73fb218 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:09.837272 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd1d5fa0-2422-458d-a708-f1f4b73fb218 admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:09.837760 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd1d5fa0-2422-458d-a708-f1f4b73fb218 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:09.838127 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd1d5fa0-2422-458d-a708-f1f4b73fb218 admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:09.838917 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd1d5fa0-2422-458d-a708-f1f4b73fb218 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:09.841068 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd1d5fa0-2422-458d-a708-f1f4b73fb218 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:09.859163 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 853/1730] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:56:09 2022] GET /identity/v3/roles => generated 1420 bytes in 62 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:56:09.861563 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd1d5fa0-2422-458d-a708-f1f4b73fb218 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:09.877173 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd1d5fa0-2422-458d-a708-f1f4b73fb218 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:09.885637 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75fd217d-5d5c-48ca-9780-9ea72713a0a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:09.886539 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75fd217d-5d5c-48ca-9780-9ea72713a0a4 admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:09.886932 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75fd217d-5d5c-48ca-9780-9ea72713a0a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:09.887302 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75fd217d-5d5c-48ca-9780-9ea72713a0a4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:09.888310 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75fd217d-5d5c-48ca-9780-9ea72713a0a4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:09.890689 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75fd217d-5d5c-48ca-9780-9ea72713a0a4 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:09.891343 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 878/1731] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:56:09 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 235 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:56:09.898356 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a5a4e2-eb34-455d-862d-a5d495c7b8b0 service nova] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:09.953636 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a5a4e2-eb34-455d-862d-a5d495c7b8b0 service nova] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:09.961746 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 854/1732] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 00:56:09 2022] POST /identity/v3/projects => generated 377 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:56:09.964125 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75fd217d-5d5c-48ca-9780-9ea72713a0a4 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:10.159446 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75fd217d-5d5c-48ca-9780-9ea72713a0a4 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:10.225045 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30dec5e2-7667-4126-9798-e7215657fe3b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:10.238139 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30dec5e2-7667-4126-9798-e7215657fe3b admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:10.238684 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30dec5e2-7667-4126-9798-e7215657fe3b admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:10.239103 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30dec5e2-7667-4126-9798-e7215657fe3b admin admin] PATH_INFO: `/v3/users` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:10.241480 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30dec5e2-7667-4126-9798-e7215657fe3b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:10.248432 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30dec5e2-7667-4126-9798-e7215657fe3b admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:10.326901 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 855/1733] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:56:09 2022] POST /identity/v3/users => generated 289 bytes in 364 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:56:10.327474 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb91b1b8-0888-4234-b42e-34846bb12e4f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:10.328290 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30dec5e2-7667-4126-9798-e7215657fe3b admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:10.330211 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb91b1b8-0888-4234-b42e-34846bb12e4f admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:10.330666 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb91b1b8-0888-4234-b42e-34846bb12e4f admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:10.331043 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb91b1b8-0888-4234-b42e-34846bb12e4f admin admin] PATH_INFO: `/v3/projects/900310e660e24f5f9957164068179944/users/f3e42870e73c497b9bcae7ef3452f8fb/roles/588016933bab436d8b7c3ac76827356d` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:10.366568 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30dec5e2-7667-4126-9798-e7215657fe3b admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:10.388179 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb91b1b8-0888-4234-b42e-34846bb12e4f admin admin] RBAC: Authorizing `identity:create_grant(project_id=900310e660e24f5f9957164068179944, user_id=f3e42870e73c497b9bcae7ef3452f8fb, role_id=588016933bab436d8b7c3ac76827356d)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:10.396935 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1bd6287-b370-44f4-a30a-35cc3e835150 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:10.398834 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb91b1b8-0888-4234-b42e-34846bb12e4f admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:10.410541 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1bd6287-b370-44f4-a30a-35cc3e835150 admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:10.417368 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1bd6287-b370-44f4-a30a-35cc3e835150 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:10.417890 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1bd6287-b370-44f4-a30a-35cc3e835150 admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:10.419341 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1bd6287-b370-44f4-a30a-35cc3e835150 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:10.430390 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1bd6287-b370-44f4-a30a-35cc3e835150 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:10.445460 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 879/1734] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:56:09 2022] PUT /identity/v3/projects/900310e660e24f5f9957164068179944/users/f3e42870e73c497b9bcae7ef3452f8fb/roles/588016933bab436d8b7c3ac76827356d => generated 0 bytes in 553 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:56:10.464856 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb4fc81-2155-46b0-b171-a9a303746a3b None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:10.465575 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb4fc81-2155-46b0-b171-a9a303746a3b None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:10.466082 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb4fc81-2155-46b0-b171-a9a303746a3b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:10.483532 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 856/1735] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:56:10 2022] GET /identity/v3/roles => generated 1420 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:56:10.487364 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1bd6287-b370-44f4-a30a-35cc3e835150 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:10.535290 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1bd6287-b370-44f4-a30a-35cc3e835150 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:10.576745 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-ceb4fc81-2155-46b0-b171-a9a303746a3b None None] MFA Rules not processed for user `f3e42870e73c497b9bcae7ef3452f8fb`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:56:10.583134 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-ceb4fc81-2155-46b0-b171-a9a303746a3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:10.630168 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:56:10.797586 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa539f18-5eb0-4323-8ddf-f7afdfaf0c8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:10.799292 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa539f18-5eb0-4323-8ddf-f7afdfaf0c8e admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:10.800286 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa539f18-5eb0-4323-8ddf-f7afdfaf0c8e admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:10.800597 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa539f18-5eb0-4323-8ddf-f7afdfaf0c8e admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:10.803713 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa539f18-5eb0-4323-8ddf-f7afdfaf0c8e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:10.808124 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa539f18-5eb0-4323-8ddf-f7afdfaf0c8e admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:10.873351 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 857/1736] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:56:10 2022] GET /identity/v3/roles => generated 1420 bytes in 387 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:56:10.874664 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa539f18-5eb0-4323-8ddf-f7afdfaf0c8e admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:10.877380 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 880/1737] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:56:10 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 414 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:56:10.880220 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceb4fc81-2155-46b0-b171-a9a303746a3b None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:10.910355 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa539f18-5eb0-4323-8ddf-f7afdfaf0c8e admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:10.925439 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceb4fc81-2155-46b0-b171-a9a303746a3b None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:10.951416 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-029a0216-73b0-4e9f-8f33-d8168d31adab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:10.953410 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029a0216-73b0-4e9f-8f33-d8168d31adab admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:10.953597 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fa134b5-c1b3-4713-ba43-705a0b443ecf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:10.954280 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa134b5-c1b3-4713-ba43-705a0b443ecf admin admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:10.954656 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa134b5-c1b3-4713-ba43-705a0b443ecf admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:10.954943 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa134b5-c1b3-4713-ba43-705a0b443ecf admin admin] PATH_INFO: `/v3/projects/9619db18b00c4670a6bb50e4d3b982ef/users/f4d29c290aa048f591948a8915e802fd/roles/588016933bab436d8b7c3ac76827356d` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:10.957360 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029a0216-73b0-4e9f-8f33-d8168d31adab admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:10.957814 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029a0216-73b0-4e9f-8f33-d8168d31adab admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:10.960807 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-029a0216-73b0-4e9f-8f33-d8168d31adab admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:10.971758 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-029a0216-73b0-4e9f-8f33-d8168d31adab admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:11.017748 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fa134b5-c1b3-4713-ba43-705a0b443ecf admin admin] RBAC: Authorizing `identity:create_grant(project_id=9619db18b00c4670a6bb50e4d3b982ef, user_id=f4d29c290aa048f591948a8915e802fd, role_id=588016933bab436d8b7c3ac76827356d)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:11.024372 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fa134b5-c1b3-4713-ba43-705a0b443ecf admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:11.025355 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:56:11.069612 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 858/1738] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:56:10 2022] PUT /identity/v3/projects/9619db18b00c4670a6bb50e4d3b982ef/users/f4d29c290aa048f591948a8915e802fd/roles/588016933bab436d8b7c3ac76827356d => generated 0 bytes in 195 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:56:11.085963 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d7b01c-b7f9-4e2c-81ee-9236813912d8 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:11.086302 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d7b01c-b7f9-4e2c-81ee-9236813912d8 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:11.086584 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d7b01c-b7f9-4e2c-81ee-9236813912d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:11.178394 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 881/1739] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 00:56:10 2022] POST /identity/v3/projects => generated 365 bytes in 300 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:56:11.179686 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-029a0216-73b0-4e9f-8f33-d8168d31adab admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:11.224052 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-63d7b01c-b7f9-4e2c-81ee-9236813912d8 None None] MFA Rules not processed for user `f4d29c290aa048f591948a8915e802fd`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:56:11.225378 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-63d7b01c-b7f9-4e2c-81ee-9236813912d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:11.417295 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-029a0216-73b0-4e9f-8f33-d8168d31adab admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:11.434427 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7d8b1b7-94ba-4638-83f8-cf0fb29cc6e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:11.444792 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d8b1b7-94ba-4638-83f8-cf0fb29cc6e8 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:11.447692 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d8b1b7-94ba-4638-83f8-cf0fb29cc6e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:11.448214 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d8b1b7-94ba-4638-83f8-cf0fb29cc6e8 admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:11.449055 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7d8b1b7-94ba-4638-83f8-cf0fb29cc6e8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:11.454048 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7d8b1b7-94ba-4638-83f8-cf0fb29cc6e8 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:11.468745 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 859/1740] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:56:11 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 390 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 09 00:56:11.475253 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63d7b01c-b7f9-4e2c-81ee-9236813912d8 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:11.511167 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 882/1741] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:56:11 2022] POST /identity/v3/users => generated 289 bytes in 332 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:56:11.512450 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63d7b01c-b7f9-4e2c-81ee-9236813912d8 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:11.515489 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7d8b1b7-94ba-4638-83f8-cf0fb29cc6e8 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:11.528650 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14516c53-802b-4c33-abab-2a0389e5bed3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:11.531706 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7d8b1b7-94ba-4638-83f8-cf0fb29cc6e8 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:11.533067 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14516c53-802b-4c33-abab-2a0389e5bed3 admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:11.533403 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14516c53-802b-4c33-abab-2a0389e5bed3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:11.534215 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14516c53-802b-4c33-abab-2a0389e5bed3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:11.536087 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14516c53-802b-4c33-abab-2a0389e5bed3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:11.548594 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14516c53-802b-4c33-abab-2a0389e5bed3 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:11.552059 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5c847f8-3576-4c98-b606-1cc449bd18e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:11.552942 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c847f8-3576-4c98-b606-1cc449bd18e8 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:11.553636 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c847f8-3576-4c98-b606-1cc449bd18e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:11.554111 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c847f8-3576-4c98-b606-1cc449bd18e8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:11.555087 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5c847f8-3576-4c98-b606-1cc449bd18e8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:11.557171 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5c847f8-3576-4c98-b606-1cc449bd18e8 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:11.583337 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 883/1742] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:56:11 2022] GET /identity/v3/roles => generated 1420 bytes in 68 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:56:11.587242 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5c847f8-3576-4c98-b606-1cc449bd18e8 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:11.621264 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5c847f8-3576-4c98-b606-1cc449bd18e8 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:11.638240 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82345064-d642-4ef1-a7f1-87ec0383822b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:11.640002 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82345064-d642-4ef1-a7f1-87ec0383822b admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:11.640949 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82345064-d642-4ef1-a7f1-87ec0383822b admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:11.642786 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82345064-d642-4ef1-a7f1-87ec0383822b admin admin] PATH_INFO: `/v3/projects/7415bcf3ee6f426ba3f908b200ad3bb3/users/9f56437d482e4a98ac03c132a300a31e/roles/092036ee46ac427baf4d56468575f697` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:11.648038 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 860/1743] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 00:56:11 2022] POST /identity/v3/projects => generated 377 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:56:11.651906 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14516c53-802b-4c33-abab-2a0389e5bed3 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:11.713869 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82345064-d642-4ef1-a7f1-87ec0383822b admin admin] RBAC: Authorizing `identity:create_grant(project_id=7415bcf3ee6f426ba3f908b200ad3bb3, user_id=9f56437d482e4a98ac03c132a300a31e, role_id=092036ee46ac427baf4d56468575f697)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:11.735242 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82345064-d642-4ef1-a7f1-87ec0383822b admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:11.794249 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 884/1744] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:56:11 2022] PUT /identity/v3/projects/7415bcf3ee6f426ba3f908b200ad3bb3/users/9f56437d482e4a98ac03c132a300a31e/roles/092036ee46ac427baf4d56468575f697 => generated 0 bytes in 206 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 09 00:56:11.798053 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82345064-d642-4ef1-a7f1-87ec0383822b admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:11.869864 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14516c53-802b-4c33-abab-2a0389e5bed3 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:11.899213 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:56:12.145012 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82345064-d642-4ef1-a7f1-87ec0383822b admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:12.184209 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bb76b01-3d0e-4e62-a424-5e9cd91c0a99 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:12.188409 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb76b01-3d0e-4e62-a424-5e9cd91c0a99 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:12.189414 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb76b01-3d0e-4e62-a424-5e9cd91c0a99 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:12.189704 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb76b01-3d0e-4e62-a424-5e9cd91c0a99 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:12.192181 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bb76b01-3d0e-4e62-a424-5e9cd91c0a99 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:12.198063 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bb76b01-3d0e-4e62-a424-5e9cd91c0a99 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:12.219397 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1603579d-2278-4328-af98-f927c9412aad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:12.220748 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1603579d-2278-4328-af98-f927c9412aad admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:12.221060 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1603579d-2278-4328-af98-f927c9412aad admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:12.221336 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1603579d-2278-4328-af98-f927c9412aad admin admin] PATH_INFO: `/v3/users` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:12.221983 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1603579d-2278-4328-af98-f927c9412aad admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:12.232163 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1603579d-2278-4328-af98-f927c9412aad admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:12.253247 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 885/1745] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:56:11 2022] GET /identity/v3/roles => generated 1420 bytes in 456 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:56:12.254918 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bb76b01-3d0e-4e62-a424-5e9cd91c0a99 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:12.339574 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 861/1746] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:56:11 2022] POST /identity/v3/users => generated 295 bytes in 685 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:56:12.342279 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bb76b01-3d0e-4e62-a424-5e9cd91c0a99 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:12.345773 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1603579d-2278-4328-af98-f927c9412aad admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:12.366643 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83681d71-e41d-47c1-ad93-438215a6ad23 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:12.367420 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83681d71-e41d-47c1-ad93-438215a6ad23 admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:12.367863 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83681d71-e41d-47c1-ad93-438215a6ad23 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:12.368176 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83681d71-e41d-47c1-ad93-438215a6ad23 admin admin] PATH_INFO: `/v3/projects/7415bcf3ee6f426ba3f908b200ad3bb3/users/9f56437d482e4a98ac03c132a300a31e/roles/092036ee46ac427baf4d56468575f697` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:12.371654 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1603579d-2278-4328-af98-f927c9412aad admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:12.376102 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83681d71-e41d-47c1-ad93-438215a6ad23 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7415bcf3ee6f426ba3f908b200ad3bb3, user_id=9f56437d482e4a98ac03c132a300a31e, role_id=092036ee46ac427baf4d56468575f697)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:12.379215 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83681d71-e41d-47c1-ad93-438215a6ad23 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:12.410617 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d5c74e2-f9e8-4acf-8d3c-eb86d6beb2f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:12.411519 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d5c74e2-f9e8-4acf-8d3c-eb86d6beb2f1 admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:12.411858 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d5c74e2-f9e8-4acf-8d3c-eb86d6beb2f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:12.412224 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d5c74e2-f9e8-4acf-8d3c-eb86d6beb2f1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:12.412869 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d5c74e2-f9e8-4acf-8d3c-eb86d6beb2f1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:12.413759 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 886/1747] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:56:12 2022] PUT /identity/v3/projects/7415bcf3ee6f426ba3f908b200ad3bb3/users/9f56437d482e4a98ac03c132a300a31e/roles/092036ee46ac427baf4d56468575f697 => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:56:12.414598 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d5c74e2-f9e8-4acf-8d3c-eb86d6beb2f1 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:12.424884 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-530d50c1-fcfb-4faa-97ed-23ca529f1ba6 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:12.425470 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-530d50c1-fcfb-4faa-97ed-23ca529f1ba6 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:12.425881 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-530d50c1-fcfb-4faa-97ed-23ca529f1ba6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:12.454143 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 862/1748] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:56:12 2022] GET /identity/v3/roles => generated 1420 bytes in 109 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:56:12.455972 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d5c74e2-f9e8-4acf-8d3c-eb86d6beb2f1 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:12.480326 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d5c74e2-f9e8-4acf-8d3c-eb86d6beb2f1 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:12.482168 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-530d50c1-fcfb-4faa-97ed-23ca529f1ba6 None None] MFA Rules not processed for user `9f56437d482e4a98ac03c132a300a31e`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:56:12.483489 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-530d50c1-fcfb-4faa-97ed-23ca529f1ba6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:12.509688 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:56:12.588133 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2517fda9-368b-4aa0-8c37-d0f28d73e306 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:12.589736 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2517fda9-368b-4aa0-8c37-d0f28d73e306 admin admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:12.590496 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2517fda9-368b-4aa0-8c37-d0f28d73e306 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:12.592669 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2517fda9-368b-4aa0-8c37-d0f28d73e306 admin admin] PATH_INFO: `/v3/projects/63fe6805f72c48ef817bef1b2cf2a37b/users/243bcff947fc4fd59e32dbc9cb272868/roles/092036ee46ac427baf4d56468575f697` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:12.639496 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2517fda9-368b-4aa0-8c37-d0f28d73e306 admin admin] RBAC: Authorizing `identity:create_grant(project_id=63fe6805f72c48ef817bef1b2cf2a37b, user_id=243bcff947fc4fd59e32dbc9cb272868, role_id=092036ee46ac427baf4d56468575f697)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:12.643733 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2517fda9-368b-4aa0-8c37-d0f28d73e306 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:12.694105 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 887/1749] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:56:12 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 267 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:56:12.695007 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 863/1750] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:56:12 2022] PUT /identity/v3/projects/63fe6805f72c48ef817bef1b2cf2a37b/users/243bcff947fc4fd59e32dbc9cb272868/roles/092036ee46ac427baf4d56468575f697 => generated 0 bytes in 240 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:56:12.699106 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-530d50c1-fcfb-4faa-97ed-23ca529f1ba6 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:12.711028 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2517fda9-368b-4aa0-8c37-d0f28d73e306 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:12.715349 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-2517fda9-368b-4aa0-8c37-d0f28d73e306 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:12.716722 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-530d50c1-fcfb-4faa-97ed-23ca529f1ba6 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:12.882196 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fb0e987-df8f-4884-8d80-a276a5e91b1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:12.885514 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fb0e987-df8f-4884-8d80-a276a5e91b1e admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:12.886485 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fb0e987-df8f-4884-8d80-a276a5e91b1e admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:12.887924 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fb0e987-df8f-4884-8d80-a276a5e91b1e admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:12.888768 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fb0e987-df8f-4884-8d80-a276a5e91b1e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:12.894171 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fb0e987-df8f-4884-8d80-a276a5e91b1e admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:12.929194 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 888/1751] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:56:12 2022] GET /identity/v3/roles => generated 1420 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:56:12.932307 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fb0e987-df8f-4884-8d80-a276a5e91b1e admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:12.989705 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fb0e987-df8f-4884-8d80-a276a5e91b1e admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:13.000410 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee578363-bf6b-4ab8-9bc8-314e4d4c61dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:13.002023 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee578363-bf6b-4ab8-9bc8-314e4d4c61dd admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:13.002538 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee578363-bf6b-4ab8-9bc8-314e4d4c61dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:13.003035 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee578363-bf6b-4ab8-9bc8-314e4d4c61dd admin admin] PATH_INFO: `/v3/projects/63fe6805f72c48ef817bef1b2cf2a37b/users/243bcff947fc4fd59e32dbc9cb272868/roles/092036ee46ac427baf4d56468575f697` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:13.009179 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee578363-bf6b-4ab8-9bc8-314e4d4c61dd admin admin] RBAC: Authorizing `identity:create_grant(project_id=63fe6805f72c48ef817bef1b2cf2a37b, user_id=243bcff947fc4fd59e32dbc9cb272868, role_id=092036ee46ac427baf4d56468575f697)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:13.018274 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee578363-bf6b-4ab8-9bc8-314e4d4c61dd admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:13.067405 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 889/1752] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:56:12 2022] PUT /identity/v3/projects/63fe6805f72c48ef817bef1b2cf2a37b/users/243bcff947fc4fd59e32dbc9cb272868/roles/092036ee46ac427baf4d56468575f697 => generated 0 bytes in 138 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:56:13.092838 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0467a92b-99f5-41f7-8200-58c0f68fa911 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:13.093260 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0467a92b-99f5-41f7-8200-58c0f68fa911 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:13.093609 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0467a92b-99f5-41f7-8200-58c0f68fa911 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:13.097141 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2517fda9-368b-4aa0-8c37-d0f28d73e306 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:13.160790 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-0467a92b-99f5-41f7-8200-58c0f68fa911 None None] MFA Rules not processed for user `243bcff947fc4fd59e32dbc9cb272868`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:56:13.162226 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-0467a92b-99f5-41f7-8200-58c0f68fa911 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:13.267922 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:56:13.418456 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 890/1753] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:56:13 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 330 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:56:13.419150 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0467a92b-99f5-41f7-8200-58c0f68fa911 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:13.429108 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-0467a92b-99f5-41f7-8200-58c0f68fa911 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:13.459436 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a83ed79-6140-4d3b-9f4c-7a43036c946f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3f48335668a4e8f934ac11ff66cd308', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:13.471015 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a83ed79-6140-4d3b-9f4c-7a43036c946f service cinder] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:13.472967 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a83ed79-6140-4d3b-9f4c-7a43036c946f service cinder] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:13.476085 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a83ed79-6140-4d3b-9f4c-7a43036c946f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:13.505127 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:56:13.652546 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0467a92b-99f5-41f7-8200-58c0f68fa911 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:13.671016 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a097e380-6128-459b-8d39-bfb290b3b49d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:13.671676 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a097e380-6128-459b-8d39-bfb290b3b49d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:13.671959 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a097e380-6128-459b-8d39-bfb290b3b49d admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:13.672237 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a097e380-6128-459b-8d39-bfb290b3b49d admin admin] PATH_INFO: `/v3/projects/46dad7bb16e340d9b0e65ac40a615b91` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:13.673831 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a097e380-6128-459b-8d39-bfb290b3b49d admin admin] RBAC: Authorizing `identity:delete_project(project_id=46dad7bb16e340d9b0e65ac40a615b91)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:13.684104 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a097e380-6128-459b-8d39-bfb290b3b49d admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:13.786618 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a83ed79-6140-4d3b-9f4c-7a43036c946f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:13.788160 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a83ed79-6140-4d3b-9f4c-7a43036c946f service cinder] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:13.805516 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 864/1754] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:56:12 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 1095 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:56:13.806842 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555c486f-2e0b-40be-9114-bc7e3575484a None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:13.807176 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555c486f-2e0b-40be-9114-bc7e3575484a None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:13.807407 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555c486f-2e0b-40be-9114-bc7e3575484a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:13.870377 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-555c486f-2e0b-40be-9114-bc7e3575484a None None] MFA Rules not processed for user `7d2eabf9865847ceb6e44d55126078a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:56:13.877225 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-555c486f-2e0b-40be-9114-bc7e3575484a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:13.980964 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-a097e380-6128-459b-8d39-bfb290b3b49d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '46dad7bb16e340d9b0e65ac40a615b91', 'request_id': 'req-a097e380-6128-459b-8d39-bfb290b3b49d'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:14.009373 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-a097e380-6128-459b-8d39-bfb290b3b49d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '46dad7bb16e340d9b0e65ac40a615b91', 'request_id': 'req-a097e380-6128-459b-8d39-bfb290b3b49d'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:14.030422 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-a097e380-6128-459b-8d39-bfb290b3b49d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '46dad7bb16e340d9b0e65ac40a615b91', 'request_id': 'req-a097e380-6128-459b-8d39-bfb290b3b49d'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:14.081850 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-a097e380-6128-459b-8d39-bfb290b3b49d admin admin] The token cache is being invalidate because project 46dad7bb16e340d9b0e65ac40a615b91 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 00:56:14.082154 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-a097e380-6128-459b-8d39-bfb290b3b49d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:14.089435 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-a097e380-6128-459b-8d39-bfb290b3b49d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:14.102154 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 891/1755] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 00:56:13 2022] DELETE /identity/v3/projects/46dad7bb16e340d9b0e65ac40a615b91 => generated 0 bytes in 682 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:56:14.106377 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a097e380-6128-459b-8d39-bfb290b3b49d admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:14.112687 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-a097e380-6128-459b-8d39-bfb290b3b49d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:14.139426 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 865/1756] 10.222.0.60 () {58 vars in 1015 bytes} [Sun Oct 9 00:56:13 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 333 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:56:14.140692 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-555c486f-2e0b-40be-9114-bc7e3575484a None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:14.156976 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-555c486f-2e0b-40be-9114-bc7e3575484a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:14.198019 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-555c486f-2e0b-40be-9114-bc7e3575484a None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:14.371905 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a097e380-6128-459b-8d39-bfb290b3b49d admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:14.380360 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-328cf26e-850c-4ac4-95d6-2a1083332e9f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3f48335668a4e8f934ac11ff66cd308', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:14.381013 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328cf26e-850c-4ac4-95d6-2a1083332e9f service cinder] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:14.381280 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328cf26e-850c-4ac4-95d6-2a1083332e9f service cinder] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:14.381559 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328cf26e-850c-4ac4-95d6-2a1083332e9f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:14.388154 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-328cf26e-850c-4ac4-95d6-2a1083332e9f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:14.445587 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30e074e5-e67f-40be-a4aa-d406c654b905 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:14.447137 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e074e5-e67f-40be-a4aa-d406c654b905 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:14.447994 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e074e5-e67f-40be-a4aa-d406c654b905 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:14.458416 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e074e5-e67f-40be-a4aa-d406c654b905 admin admin] PATH_INFO: `/v3/users/70fdef2a2e194a18b5b80696a911ab1f` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:14.466066 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30e074e5-e67f-40be-a4aa-d406c654b905 admin admin] RBAC: Authorizing `identity:delete_user(user_id=70fdef2a2e194a18b5b80696a911ab1f)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:14.473763 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30e074e5-e67f-40be-a4aa-d406c654b905 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:14.528413 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-328cf26e-850c-4ac4-95d6-2a1083332e9f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:14.530013 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-328cf26e-850c-4ac4-95d6-2a1083332e9f service cinder] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:14.536430 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:56:14.544344 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 892/1757] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:56:14 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 441 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:56:14.546389 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-328cf26e-850c-4ac4-95d6-2a1083332e9f service cinder] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:14.549676 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-328cf26e-850c-4ac4-95d6-2a1083332e9f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:14.599685 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:56:14.662297 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-30e074e5-e67f-40be-a4aa-d406c654b905 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '70fdef2a2e194a18b5b80696a911ab1f'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:14.682774 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-30e074e5-e67f-40be-a4aa-d406c654b905 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '70fdef2a2e194a18b5b80696a911ab1f'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:14.687270 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-30e074e5-e67f-40be-a4aa-d406c654b905 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '70fdef2a2e194a18b5b80696a911ab1f'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:14.715268 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-30e074e5-e67f-40be-a4aa-d406c654b905 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '70fdef2a2e194a18b5b80696a911ab1f'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:14.759941 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-30e074e5-e67f-40be-a4aa-d406c654b905 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '70fdef2a2e194a18b5b80696a911ab1f'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:14.771175 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-328cf26e-850c-4ac4-95d6-2a1083332e9f service cinder] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:14.782816 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-472ed91f-f368-4c4e-ac32-95a52894a664 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:14.783793 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-472ed91f-f368-4c4e-ac32-95a52894a664 service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:14.784160 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-472ed91f-f368-4c4e-ac32-95a52894a664 service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:14.784520 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-472ed91f-f368-4c4e-ac32-95a52894a664 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:14.789607 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-472ed91f-f368-4c4e-ac32-95a52894a664 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:14.815325 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-472ed91f-f368-4c4e-ac32-95a52894a664 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:14.817061 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-472ed91f-f368-4c4e-ac32-95a52894a664 service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:14.828822 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 866/1758] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 00:56:14 2022] DELETE /identity/v3/users/70fdef2a2e194a18b5b80696a911ab1f => generated 0 bytes in 666 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:56:14.837327 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30e074e5-e67f-40be-a4aa-d406c654b905 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:14.844748 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-30e074e5-e67f-40be-a4aa-d406c654b905 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:14.849000 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 893/1759] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:56:14 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 296 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:56:14.851097 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e1459e-1f59-4005-8271-baf1bb0bacee None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:14.853901 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e1459e-1f59-4005-8271-baf1bb0bacee None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:14.856391 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e1459e-1f59-4005-8271-baf1bb0bacee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:14.950850 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-a1e1459e-1f59-4005-8271-baf1bb0bacee None None] MFA Rules not processed for user `438446d1e6b3402bb18b547c089391f8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:56:14.952997 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-a1e1459e-1f59-4005-8271-baf1bb0bacee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:15.150350 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30e074e5-e67f-40be-a4aa-d406c654b905 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:15.185958 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47143cdc-9362-424c-b088-1386350b764c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:15.186608 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47143cdc-9362-424c-b088-1386350b764c service neutron] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:15.186948 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47143cdc-9362-424c-b088-1386350b764c service neutron] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:15.187160 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47143cdc-9362-424c-b088-1386350b764c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:15.187640 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 894/1760] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:56:14 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 338 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:56:15.189964 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1e1459e-1f59-4005-8271-baf1bb0bacee None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:15.196412 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-a1e1459e-1f59-4005-8271-baf1bb0bacee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:15.222220 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47143cdc-9362-424c-b088-1386350b764c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:15.227326 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47143cdc-9362-424c-b088-1386350b764c service neutron] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:15.270468 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 867/1761] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:56:14 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 441 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:56:15.373117 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1e1459e-1f59-4005-8271-baf1bb0bacee None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:15.386232 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38bbcba9-fcc9-495c-90e6-0c0f7f8ebad8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3f48335668a4e8f934ac11ff66cd308', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:15.389773 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38bbcba9-fcc9-495c-90e6-0c0f7f8ebad8 service cinder] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:15.390332 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38bbcba9-fcc9-495c-90e6-0c0f7f8ebad8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:15.391274 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38bbcba9-fcc9-495c-90e6-0c0f7f8ebad8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:15.397443 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-38bbcba9-fcc9-495c-90e6-0c0f7f8ebad8 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:15.537489 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38bbcba9-fcc9-495c-90e6-0c0f7f8ebad8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:15.539606 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38bbcba9-fcc9-495c-90e6-0c0f7f8ebad8 service cinder] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:15.553706 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 895/1762] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:56:15 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 365 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:56:16.726781 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47143cdc-9362-424c-b088-1386350b764c service neutron] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:16.731675 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-47143cdc-9362-424c-b088-1386350b764c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:17.088958 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47143cdc-9362-424c-b088-1386350b764c service neutron] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:17.108849 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d6732fb-12de-4ab3-a62c-ea814ad8b2cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:17.109538 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d6732fb-12de-4ab3-a62c-ea814ad8b2cb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:17.109916 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d6732fb-12de-4ab3-a62c-ea814ad8b2cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:17.110713 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d6732fb-12de-4ab3-a62c-ea814ad8b2cb admin admin] PATH_INFO: `/v3/projects/28e69a2efede4f4a834de16f5712ec2e` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:17.112731 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d6732fb-12de-4ab3-a62c-ea814ad8b2cb admin admin] RBAC: Authorizing `identity:delete_project(project_id=28e69a2efede4f4a834de16f5712ec2e)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:17.122903 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d6732fb-12de-4ab3-a62c-ea814ad8b2cb admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:17.356820 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-7d6732fb-12de-4ab3-a62c-ea814ad8b2cb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '28e69a2efede4f4a834de16f5712ec2e', 'request_id': 'req-7d6732fb-12de-4ab3-a62c-ea814ad8b2cb'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:17.361243 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:56:17.381449 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-7d6732fb-12de-4ab3-a62c-ea814ad8b2cb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '28e69a2efede4f4a834de16f5712ec2e', 'request_id': 'req-7d6732fb-12de-4ab3-a62c-ea814ad8b2cb'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:17.407741 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-7d6732fb-12de-4ab3-a62c-ea814ad8b2cb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '28e69a2efede4f4a834de16f5712ec2e', 'request_id': 'req-7d6732fb-12de-4ab3-a62c-ea814ad8b2cb'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:17.466126 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-7d6732fb-12de-4ab3-a62c-ea814ad8b2cb admin admin] The token cache is being invalidate because project 28e69a2efede4f4a834de16f5712ec2e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81453) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 00:56:17.466523 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-7d6732fb-12de-4ab3-a62c-ea814ad8b2cb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:17.468589 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-7d6732fb-12de-4ab3-a62c-ea814ad8b2cb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:17.477644 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 868/1763] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 00:56:16 2022] DELETE /identity/v3/projects/28e69a2efede4f4a834de16f5712ec2e => generated 0 bytes in 748 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:56:17.479766 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-578fe1cd-b398-427d-8345-b37a6acc8980 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:17.480705 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-578fe1cd-b398-427d-8345-b37a6acc8980 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:17.481173 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-578fe1cd-b398-427d-8345-b37a6acc8980 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:17.496811 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:56:17.545782 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-578fe1cd-b398-427d-8345-b37a6acc8980 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:56:17.551550 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-578fe1cd-b398-427d-8345-b37a6acc8980 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:17.743867 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 896/1764] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:56:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 266 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:56:17.754945 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afe6edcd-2972-4c3d-b720-f46e707c2c59 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:17.755501 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afe6edcd-2972-4c3d-b720-f46e707c2c59 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:17.755501 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afe6edcd-2972-4c3d-b720-f46e707c2c59 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:17.808909 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-afe6edcd-2972-4c3d-b720-f46e707c2c59 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:56:17.810186 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-afe6edcd-2972-4c3d-b720-f46e707c2c59 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:17.838825 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-578fe1cd-b398-427d-8345-b37a6acc8980 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:17.853805 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 869/1765] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:56:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:56:17.860596 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-578fe1cd-b398-427d-8345-b37a6acc8980 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:17.870168 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1044bbe7-237c-4e7b-9652-3374863b62d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:17.871341 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1044bbe7-237c-4e7b-9652-3374863b62d3 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:17.872547 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1044bbe7-237c-4e7b-9652-3374863b62d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:17.872844 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1044bbe7-237c-4e7b-9652-3374863b62d3 admin admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:17.874217 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1044bbe7-237c-4e7b-9652-3374863b62d3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:56:17.874900 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1044bbe7-237c-4e7b-9652-3374863b62d3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:17.879110 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1044bbe7-237c-4e7b-9652-3374863b62d3 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:17.938059 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 897/1766] 10.222.0.33 () {58 vars in 1153 bytes} [Sun Oct 9 00:56:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 102 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:56:17.942655 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afe6edcd-2972-4c3d-b720-f46e707c2c59 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:17.964927 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afe6edcd-2972-4c3d-b720-f46e707c2c59 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:17.985141 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ffde43a-756b-404f-b444-c9ba2469bddc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:17.985141 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ffde43a-756b-404f-b444-c9ba2469bddc admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:17.985421 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ffde43a-756b-404f-b444-c9ba2469bddc admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:17.985725 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ffde43a-756b-404f-b444-c9ba2469bddc admin admin] PATH_INFO: `/v3/projects` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:17.986382 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ffde43a-756b-404f-b444-c9ba2469bddc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:17.988364 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ffde43a-756b-404f-b444-c9ba2469bddc admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:18.101271 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 870/1767] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 00:56:17 2022] POST /identity/v3/projects => generated 357 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:56:18.105259 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1044bbe7-237c-4e7b-9652-3374863b62d3 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:18.118940 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1044bbe7-237c-4e7b-9652-3374863b62d3 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:18.320727 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e6ca2f0-bfc7-43a5-a603-47ef490dd92b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:18.321034 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e6ca2f0-bfc7-43a5-a603-47ef490dd92b admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:18.321320 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e6ca2f0-bfc7-43a5-a603-47ef490dd92b admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:18.321595 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e6ca2f0-bfc7-43a5-a603-47ef490dd92b admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:18.322238 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e6ca2f0-bfc7-43a5-a603-47ef490dd92b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:18.333017 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e6ca2f0-bfc7-43a5-a603-47ef490dd92b admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:18.397242 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 898/1768] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:56:18 2022] POST /identity/v3/users => generated 279 bytes in 292 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:56:18.401530 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ffde43a-756b-404f-b444-c9ba2469bddc admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:18.418977 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ffde43a-756b-404f-b444-c9ba2469bddc admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:18.433556 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca694831-0c93-4207-8b19-2cb2a7fb007d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:18.434482 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca694831-0c93-4207-8b19-2cb2a7fb007d admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:18.434937 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca694831-0c93-4207-8b19-2cb2a7fb007d admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:18.435369 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca694831-0c93-4207-8b19-2cb2a7fb007d admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:18.436810 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca694831-0c93-4207-8b19-2cb2a7fb007d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:18.439464 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca694831-0c93-4207-8b19-2cb2a7fb007d admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:18.462650 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e6ca2f0-bfc7-43a5-a603-47ef490dd92b admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:18.463332 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 871/1769] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:56:18 2022] GET /identity/v3/roles => generated 1420 bytes in 60 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:56:18.474897 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e6ca2f0-bfc7-43a5-a603-47ef490dd92b admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:18.480358 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6124b295-480d-4a6b-b409-cd06576a87a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:18.481706 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6124b295-480d-4a6b-b409-cd06576a87a5 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:18.482155 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6124b295-480d-4a6b-b409-cd06576a87a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:18.482580 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6124b295-480d-4a6b-b409-cd06576a87a5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:18.483739 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6124b295-480d-4a6b-b409-cd06576a87a5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:18.485574 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6124b295-480d-4a6b-b409-cd06576a87a5 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:18.502623 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca694831-0c93-4207-8b19-2cb2a7fb007d admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:18.510040 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 899/1770] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:56:18 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:56:18.520051 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca694831-0c93-4207-8b19-2cb2a7fb007d admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:18.531710 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54d9303b-3ec4-48e2-846a-8f2c5137c335 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:18.532502 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d9303b-3ec4-48e2-846a-8f2c5137c335 admin admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:18.533737 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d9303b-3ec4-48e2-846a-8f2c5137c335 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:18.534053 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d9303b-3ec4-48e2-846a-8f2c5137c335 admin admin] PATH_INFO: `/v3/projects/2912461a18214cb5a7b0762205e64b1e/users/e4848d52d9384834a7946936c029c456/roles/588016933bab436d8b7c3ac76827356d` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:18.554673 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54d9303b-3ec4-48e2-846a-8f2c5137c335 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2912461a18214cb5a7b0762205e64b1e, user_id=e4848d52d9384834a7946936c029c456, role_id=588016933bab436d8b7c3ac76827356d)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:18.557465 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54d9303b-3ec4-48e2-846a-8f2c5137c335 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:18.576682 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 872/1771] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:56:18 2022] PUT /identity/v3/projects/2912461a18214cb5a7b0762205e64b1e/users/e4848d52d9384834a7946936c029c456/roles/588016933bab436d8b7c3ac76827356d => generated 0 bytes in 75 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:56:18.589472 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd247079-1d7c-4273-a2f9-69d13f669923 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:18.589772 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd247079-1d7c-4273-a2f9-69d13f669923 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:18.590029 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd247079-1d7c-4273-a2f9-69d13f669923 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:18.639748 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-dd247079-1d7c-4273-a2f9-69d13f669923 None None] MFA Rules not processed for user `e4848d52d9384834a7946936c029c456`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:56:18.641562 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-dd247079-1d7c-4273-a2f9-69d13f669923 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:18.761618 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:56:18.782226 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 900/1772] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:56:18 2022] POST /identity/v3/auth/tokens => generated 3234 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 09 00:56:18.784649 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54d9303b-3ec4-48e2-846a-8f2c5137c335 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:18.806450 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e35d331-fbf0-4512-a7b4-34111c82cbe9 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:18.806922 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e35d331-fbf0-4512-a7b4-34111c82cbe9 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:18.807205 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e35d331-fbf0-4512-a7b4-34111c82cbe9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:18.857229 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-0e35d331-fbf0-4512-a7b4-34111c82cbe9 None None] MFA Rules not processed for user `b027fb8adadc488e975c4541e7ee6626`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:56:18.863392 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-0e35d331-fbf0-4512-a7b4-34111c82cbe9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:18.954180 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54d9303b-3ec4-48e2-846a-8f2c5137c335 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:18.969521 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d07782f4-99eb-4782-b636-1b3486e1b8d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:18.970683 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07782f4-99eb-4782-b636-1b3486e1b8d5 admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:18.970964 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07782f4-99eb-4782-b636-1b3486e1b8d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:18.971238 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07782f4-99eb-4782-b636-1b3486e1b8d5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:18.971909 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d07782f4-99eb-4782-b636-1b3486e1b8d5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:18.974786 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d07782f4-99eb-4782-b636-1b3486e1b8d5 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:18.997759 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 901/1773] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:56:18 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:56:18.998984 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f39954-63b2-4b37-8dbf-095a757dc678 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:18.999273 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f39954-63b2-4b37-8dbf-095a757dc678 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:18.999541 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f39954-63b2-4b37-8dbf-095a757dc678 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:19.000843 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 902/1774] 10.222.0.33 () {54 vars in 921 bytes} [Sun Oct 9 00:56:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:56:19.030831 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 873/1775] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 00:56:18 2022] POST /identity/v3/projects => generated 357 bytes in 244 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:56:19.031963 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80f39954-63b2-4b37-8dbf-095a757dc678 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:19.052709 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44063265-0f13-491c-9401-7d4115aaf079 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:19.052835 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44063265-0f13-491c-9401-7d4115aaf079 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:19.053079 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44063265-0f13-491c-9401-7d4115aaf079 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:19.102501 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-44063265-0f13-491c-9401-7d4115aaf079 None None] MFA Rules not processed for user `e0a5d95969e74157bb2d2a57207e1651`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:56:19.105856 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-44063265-0f13-491c-9401-7d4115aaf079 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:19.196935 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:DELETE=3,keystone:SELECT=93,keystone:INSERT=4 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:56:19.211820 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80f39954-63b2-4b37-8dbf-095a757dc678 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:19.217513 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5f76412-07fa-40f3-b3e4-2657a78d0dac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:19.218122 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5f76412-07fa-40f3-b3e4-2657a78d0dac admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:19.218400 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5f76412-07fa-40f3-b3e4-2657a78d0dac admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:19.218671 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5f76412-07fa-40f3-b3e4-2657a78d0dac admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:19.219303 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5f76412-07fa-40f3-b3e4-2657a78d0dac admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:19.221965 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5f76412-07fa-40f3-b3e4-2657a78d0dac admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:19.224373 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 874/1776] 10.222.0.33 () {58 vars in 1018 bytes} [Sun Oct 9 00:56:19 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:56:19.269107 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cca911e-7756-40ac-835b-1c67e1d32798 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:19.269574 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cca911e-7756-40ac-835b-1c67e1d32798 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:19.270002 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cca911e-7756-40ac-835b-1c67e1d32798 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:19.273295 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 875/1777] 10.222.0.33 () {54 vars in 921 bytes} [Sun Oct 9 00:56:19 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:56:19.279948 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 903/1778] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:56:19 2022] POST /identity/v3/users => generated 285 bytes in 248 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:56:19.284084 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cca911e-7756-40ac-835b-1c67e1d32798 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:19.299873 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d479ffc9-0d41-4d41-a9eb-69736b8c34fe None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:19.300280 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d479ffc9-0d41-4d41-a9eb-69736b8c34fe None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:19.300624 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d479ffc9-0d41-4d41-a9eb-69736b8c34fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:19.315811 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cca911e-7756-40ac-835b-1c67e1d32798 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:19.322791 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5ea372a-97fa-43bc-94fc-8c65236a45ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:19.324225 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5ea372a-97fa-43bc-94fc-8c65236a45ab admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:19.324775 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5ea372a-97fa-43bc-94fc-8c65236a45ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:19.325232 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5ea372a-97fa-43bc-94fc-8c65236a45ab admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:19.326343 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5ea372a-97fa-43bc-94fc-8c65236a45ab admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:19.328423 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5ea372a-97fa-43bc-94fc-8c65236a45ab admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:19.337167 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-d479ffc9-0d41-4d41-a9eb-69736b8c34fe None None] MFA Rules not processed for user `e0a5d95969e74157bb2d2a57207e1651`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:56:19.338385 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-d479ffc9-0d41-4d41-a9eb-69736b8c34fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:19.349350 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 876/1779] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:56:19 2022] GET /identity/v3/roles => generated 1420 bytes in 65 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:56:19.350333 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5ea372a-97fa-43bc-94fc-8c65236a45ab admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:19.362637 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5ea372a-97fa-43bc-94fc-8c65236a45ab admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:19.378495 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8b9ec86-77eb-4f63-9ff7-5026242613fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:19.379164 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8b9ec86-77eb-4f63-9ff7-5026242613fb admin admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:19.379559 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8b9ec86-77eb-4f63-9ff7-5026242613fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:19.379961 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8b9ec86-77eb-4f63-9ff7-5026242613fb admin admin] PATH_INFO: `/v3/projects/abf32db91d464fe4b470eebe5bf0c78d/users/604ccc8529494def8a7dc54747a1fd79/roles/092036ee46ac427baf4d56468575f697` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:19.396767 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 904/1780] 10.222.0.33 () {58 vars in 1018 bytes} [Sun Oct 9 00:56:19 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:56:19.409351 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d479ffc9-0d41-4d41-a9eb-69736b8c34fe None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:19.410825 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-d479ffc9-0d41-4d41-a9eb-69736b8c34fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:19.428685 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8b9ec86-77eb-4f63-9ff7-5026242613fb admin admin] RBAC: Authorizing `identity:create_grant(project_id=abf32db91d464fe4b470eebe5bf0c78d, user_id=604ccc8529494def8a7dc54747a1fd79, role_id=092036ee46ac427baf4d56468575f697)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:19.437619 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d479ffc9-0d41-4d41-a9eb-69736b8c34fe None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:19.437812 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8b9ec86-77eb-4f63-9ff7-5026242613fb admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:19.449232 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5ea7fb9-c10b-45c7-9780-47800a5f6569 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a5d95969e74157bb2d2a57207e1651', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:19.449972 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5ea7fb9-c10b-45c7-9780-47800a5f6569 service nova] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:19.450518 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5ea7fb9-c10b-45c7-9780-47800a5f6569 service nova] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:19.450919 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5ea7fb9-c10b-45c7-9780-47800a5f6569 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:19.460241 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 877/1781] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:56:19 2022] PUT /identity/v3/projects/abf32db91d464fe4b470eebe5bf0c78d/users/604ccc8529494def8a7dc54747a1fd79/roles/092036ee46ac427baf4d56468575f697 => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:56:19.467141 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5ea7fb9-c10b-45c7-9780-47800a5f6569 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:19.468496 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8b9ec86-77eb-4f63-9ff7-5026242613fb admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:19.473044 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5ea7fb9-c10b-45c7-9780-47800a5f6569 service nova] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:19.502345 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 905/1782] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:56:19 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:56:19.604941 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8b9ec86-77eb-4f63-9ff7-5026242613fb admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:19.610168 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45adc78c-aa27-4607-ad01-4344b83dacf0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:19.612361 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45adc78c-aa27-4607-ad01-4344b83dacf0 admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:19.612897 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45adc78c-aa27-4607-ad01-4344b83dacf0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:19.614122 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45adc78c-aa27-4607-ad01-4344b83dacf0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:19.616768 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45adc78c-aa27-4607-ad01-4344b83dacf0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:19.620590 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45adc78c-aa27-4607-ad01-4344b83dacf0 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:19.653615 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5ea7fb9-c10b-45c7-9780-47800a5f6569 service nova] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:19.654261 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 878/1783] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:56:19 2022] GET /identity/v3/roles => generated 1420 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:56:19.674139 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5ea7fb9-c10b-45c7-9780-47800a5f6569 service nova] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:19.681458 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-847e3529-0cc0-405f-a588-d65ef5a85ce7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:19.682123 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847e3529-0cc0-405f-a588-d65ef5a85ce7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:19.682400 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847e3529-0cc0-405f-a588-d65ef5a85ce7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:19.682670 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847e3529-0cc0-405f-a588-d65ef5a85ce7 admin admin] PATH_INFO: `/v3/projects/abf32db91d464fe4b470eebe5bf0c78d/users/604ccc8529494def8a7dc54747a1fd79/roles/092036ee46ac427baf4d56468575f697` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:19.690276 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-847e3529-0cc0-405f-a588-d65ef5a85ce7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=abf32db91d464fe4b470eebe5bf0c78d, user_id=604ccc8529494def8a7dc54747a1fd79, role_id=092036ee46ac427baf4d56468575f697)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:19.692986 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-847e3529-0cc0-405f-a588-d65ef5a85ce7 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:19.716297 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 906/1784] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:56:19 2022] PUT /identity/v3/projects/abf32db91d464fe4b470eebe5bf0c78d/users/604ccc8529494def8a7dc54747a1fd79/roles/092036ee46ac427baf4d56468575f697 => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:56:19.732193 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45b0b8cd-b654-41b8-a932-aedd282a0486 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:19.732193 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45b0b8cd-b654-41b8-a932-aedd282a0486 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:19.732193 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45b0b8cd-b654-41b8-a932-aedd282a0486 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:19.745462 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-847e3529-0cc0-405f-a588-d65ef5a85ce7 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:19.779236 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-45b0b8cd-b654-41b8-a932-aedd282a0486 None None] MFA Rules not processed for user `604ccc8529494def8a7dc54747a1fd79`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:56:19.781044 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-45b0b8cd-b654-41b8-a932-aedd282a0486 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:19.818449 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:56:19.845270 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-847e3529-0cc0-405f-a588-d65ef5a85ce7 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:19.855526 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25695ba0-8cd7-4485-a504-93bec3617243 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a5d95969e74157bb2d2a57207e1651', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:19.856168 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25695ba0-8cd7-4485-a504-93bec3617243 service nova] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:19.856449 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25695ba0-8cd7-4485-a504-93bec3617243 service nova] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:19.856719 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25695ba0-8cd7-4485-a504-93bec3617243 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:19.877778 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 879/1785] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:56:19 2022] POST /identity/v3/auth/tokens => generated 3301 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:56:19.878284 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25695ba0-8cd7-4485-a504-93bec3617243 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:19.878284 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25695ba0-8cd7-4485-a504-93bec3617243 service nova] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:19.887231 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45b0b8cd-b654-41b8-a932-aedd282a0486 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:19.888936 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-45b0b8cd-b654-41b8-a932-aedd282a0486 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:19.889731 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 907/1786] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:56:19 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 145 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:56:20.008505 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:56:20.041369 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45b0b8cd-b654-41b8-a932-aedd282a0486 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:20.051841 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-508c25f4-6612-42e5-8dc8-a246a6435717 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3f48335668a4e8f934ac11ff66cd308', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:20.052456 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-508c25f4-6612-42e5-8dc8-a246a6435717 service cinder] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:20.052937 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-508c25f4-6612-42e5-8dc8-a246a6435717 service cinder] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:20.053802 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-508c25f4-6612-42e5-8dc8-a246a6435717 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:20.074631 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-508c25f4-6612-42e5-8dc8-a246a6435717 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:20.076218 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-508c25f4-6612-42e5-8dc8-a246a6435717 service cinder] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:20.093388 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 880/1787] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:56:19 2022] GET /identity/v3/auth/tokens => generated 3301 bytes in 207 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:56:20.196558 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25695ba0-8cd7-4485-a504-93bec3617243 service nova] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:20.207611 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25695ba0-8cd7-4485-a504-93bec3617243 service nova] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:20.223619 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb4c5e13-7c71-4a28-a46d-af6b39f0b492 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3f48335668a4e8f934ac11ff66cd308', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:20.223619 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb4c5e13-7c71-4a28-a46d-af6b39f0b492 service cinder] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:20.223619 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb4c5e13-7c71-4a28-a46d-af6b39f0b492 service cinder] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:20.223619 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb4c5e13-7c71-4a28-a46d-af6b39f0b492 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:20.293722 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb4c5e13-7c71-4a28-a46d-af6b39f0b492 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:20.295993 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb4c5e13-7c71-4a28-a46d-af6b39f0b492 service cinder] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:20.306755 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 908/1788] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:56:20 2022] GET /identity/v3/auth/tokens => generated 3234 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:56:21.028477 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-508c25f4-6612-42e5-8dc8-a246a6435717 service cinder] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:21.030511 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-508c25f4-6612-42e5-8dc8-a246a6435717 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:21.053323 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-508c25f4-6612-42e5-8dc8-a246a6435717 service cinder] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:21.060373 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17170a22-b39f-435d-a5a0-58ea003e2fe4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3f48335668a4e8f934ac11ff66cd308', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:21.061239 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17170a22-b39f-435d-a5a0-58ea003e2fe4 service cinder] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:21.061709 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17170a22-b39f-435d-a5a0-58ea003e2fe4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:21.062148 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17170a22-b39f-435d-a5a0-58ea003e2fe4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:21.065772 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-17170a22-b39f-435d-a5a0-58ea003e2fe4 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:21.160163 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17170a22-b39f-435d-a5a0-58ea003e2fe4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:21.163014 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17170a22-b39f-435d-a5a0-58ea003e2fe4 service cinder] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:21.181240 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 881/1789] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:56:21 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 153 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:56:30.301875 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=35 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:56:31.169275 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=39 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:56:41.048025 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb4c5e13-7c71-4a28-a46d-af6b39f0b492 service cinder] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:41.050002 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-bb4c5e13-7c71-4a28-a46d-af6b39f0b492 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:41.135084 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb4c5e13-7c71-4a28-a46d-af6b39f0b492 service cinder] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:41.139658 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf1156d6-872b-42bc-a11f-a64bbd86ea01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:41.140294 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf1156d6-872b-42bc-a11f-a64bbd86ea01 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:41.140572 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf1156d6-872b-42bc-a11f-a64bbd86ea01 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:41.140889 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf1156d6-872b-42bc-a11f-a64bbd86ea01 admin admin] PATH_INFO: `/v3/users/f3e42870e73c497b9bcae7ef3452f8fb` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:41.141955 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf1156d6-872b-42bc-a11f-a64bbd86ea01 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f3e42870e73c497b9bcae7ef3452f8fb)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:41.144004 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf1156d6-872b-42bc-a11f-a64bbd86ea01 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:41.195562 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-bf1156d6-872b-42bc-a11f-a64bbd86ea01 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f3e42870e73c497b9bcae7ef3452f8fb'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:41.203034 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-bf1156d6-872b-42bc-a11f-a64bbd86ea01 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f3e42870e73c497b9bcae7ef3452f8fb'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:41.203724 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-bf1156d6-872b-42bc-a11f-a64bbd86ea01 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f3e42870e73c497b9bcae7ef3452f8fb'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:41.204339 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-bf1156d6-872b-42bc-a11f-a64bbd86ea01 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f3e42870e73c497b9bcae7ef3452f8fb'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:41.214009 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-bf1156d6-872b-42bc-a11f-a64bbd86ea01 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f3e42870e73c497b9bcae7ef3452f8fb'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:41.232238 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 909/1790] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 00:56:41 2022] DELETE /identity/v3/users/f3e42870e73c497b9bcae7ef3452f8fb => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:56:41.236727 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddfe000b-f432-4b15-bf96-72720e0a7af8 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:41.237532 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddfe000b-f432-4b15-bf96-72720e0a7af8 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:41.237830 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddfe000b-f432-4b15-bf96-72720e0a7af8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:41.264564 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-ddfe000b-f432-4b15-bf96-72720e0a7af8 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:56:41.265908 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-ddfe000b-f432-4b15-bf96-72720e0a7af8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:41.339766 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 882/1791] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:56:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:56:41.349448 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf1156d6-872b-42bc-a11f-a64bbd86ea01 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:41.350942 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-bf1156d6-872b-42bc-a11f-a64bbd86ea01 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:41.422622 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf1156d6-872b-42bc-a11f-a64bbd86ea01 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:41.428756 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-379534e8-8b0b-4a16-b0dd-679867fd401a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:41.428756 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-379534e8-8b0b-4a16-b0dd-679867fd401a service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:41.428756 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-379534e8-8b0b-4a16-b0dd-679867fd401a service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:41.428756 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-379534e8-8b0b-4a16-b0dd-679867fd401a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:41.442392 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-379534e8-8b0b-4a16-b0dd-679867fd401a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:41.444222 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-379534e8-8b0b-4a16-b0dd-679867fd401a service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:41.459521 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 910/1792] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:56:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:56:41.919930 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddfe000b-f432-4b15-bf96-72720e0a7af8 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:41.921570 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-ddfe000b-f432-4b15-bf96-72720e0a7af8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:41.936507 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddfe000b-f432-4b15-bf96-72720e0a7af8 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:41.942233 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d11b6b8-d8c7-40af-b19d-a05626f3adb2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:41.943169 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d11b6b8-d8c7-40af-b19d-a05626f3adb2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:41.944886 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d11b6b8-d8c7-40af-b19d-a05626f3adb2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:41.945170 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d11b6b8-d8c7-40af-b19d-a05626f3adb2 admin admin] PATH_INFO: `/v3/projects/900310e660e24f5f9957164068179944` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:41.946545 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d11b6b8-d8c7-40af-b19d-a05626f3adb2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=900310e660e24f5f9957164068179944)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:41.948727 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d11b6b8-d8c7-40af-b19d-a05626f3adb2 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:41.998253 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-0d11b6b8-d8c7-40af-b19d-a05626f3adb2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '900310e660e24f5f9957164068179944', 'request_id': 'req-0d11b6b8-d8c7-40af-b19d-a05626f3adb2'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:42.004576 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-0d11b6b8-d8c7-40af-b19d-a05626f3adb2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '900310e660e24f5f9957164068179944', 'request_id': 'req-0d11b6b8-d8c7-40af-b19d-a05626f3adb2'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:42.010948 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-0d11b6b8-d8c7-40af-b19d-a05626f3adb2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '900310e660e24f5f9957164068179944', 'request_id': 'req-0d11b6b8-d8c7-40af-b19d-a05626f3adb2'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:42.024275 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-0d11b6b8-d8c7-40af-b19d-a05626f3adb2 admin admin] The token cache is being invalidate because project 900310e660e24f5f9957164068179944 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81453) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 00:56:42.024596 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-0d11b6b8-d8c7-40af-b19d-a05626f3adb2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:42.025355 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-0d11b6b8-d8c7-40af-b19d-a05626f3adb2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:42.026862 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 883/1793] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 00:56:41 2022] DELETE /identity/v3/projects/900310e660e24f5f9957164068179944 => generated 0 bytes in 108 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:56:42.031237 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-379534e8-8b0b-4a16-b0dd-679867fd401a service neutron] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:42.041450 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-379534e8-8b0b-4a16-b0dd-679867fd401a service neutron] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:42.066256 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:56:42.121729 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-581b631e-6d2e-4c5c-9ba8-b78a5a9225cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:42.122364 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-581b631e-6d2e-4c5c-9ba8-b78a5a9225cb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:42.122644 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-581b631e-6d2e-4c5c-9ba8-b78a5a9225cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:42.123065 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-581b631e-6d2e-4c5c-9ba8-b78a5a9225cb admin admin] PATH_INFO: `/v3/users/9f56437d482e4a98ac03c132a300a31e` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:42.124420 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-581b631e-6d2e-4c5c-9ba8-b78a5a9225cb admin admin] RBAC: Authorizing `identity:delete_user(user_id=9f56437d482e4a98ac03c132a300a31e)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:42.126345 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-581b631e-6d2e-4c5c-9ba8-b78a5a9225cb admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:42.202038 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-581b631e-6d2e-4c5c-9ba8-b78a5a9225cb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9f56437d482e4a98ac03c132a300a31e'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:42.209165 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-581b631e-6d2e-4c5c-9ba8-b78a5a9225cb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9f56437d482e4a98ac03c132a300a31e'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:42.209848 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-581b631e-6d2e-4c5c-9ba8-b78a5a9225cb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9f56437d482e4a98ac03c132a300a31e'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:42.210472 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-581b631e-6d2e-4c5c-9ba8-b78a5a9225cb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9f56437d482e4a98ac03c132a300a31e'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:42.219270 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-581b631e-6d2e-4c5c-9ba8-b78a5a9225cb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9f56437d482e4a98ac03c132a300a31e'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:42.240754 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 911/1794] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 00:56:42 2022] DELETE /identity/v3/users/9f56437d482e4a98ac03c132a300a31e => generated 0 bytes in 210 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:56:42.507843 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d11b6b8-d8c7-40af-b19d-a05626f3adb2 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:42.509328 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-0d11b6b8-d8c7-40af-b19d-a05626f3adb2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:42.523380 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d11b6b8-d8c7-40af-b19d-a05626f3adb2 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:42.583464 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5618c97-9b50-4169-9481-bae1bc32be8a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:42.584239 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5618c97-9b50-4169-9481-bae1bc32be8a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:42.584522 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5618c97-9b50-4169-9481-bae1bc32be8a admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:42.584817 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5618c97-9b50-4169-9481-bae1bc32be8a admin admin] PATH_INFO: `/v3/projects/7415bcf3ee6f426ba3f908b200ad3bb3` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:42.585932 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5618c97-9b50-4169-9481-bae1bc32be8a admin admin] RBAC: Authorizing `identity:delete_project(project_id=7415bcf3ee6f426ba3f908b200ad3bb3)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:42.587999 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5618c97-9b50-4169-9481-bae1bc32be8a admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:42.620315 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:56:42.639847 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-d5618c97-9b50-4169-9481-bae1bc32be8a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7415bcf3ee6f426ba3f908b200ad3bb3', 'request_id': 'req-d5618c97-9b50-4169-9481-bae1bc32be8a'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:42.646122 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-d5618c97-9b50-4169-9481-bae1bc32be8a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7415bcf3ee6f426ba3f908b200ad3bb3', 'request_id': 'req-d5618c97-9b50-4169-9481-bae1bc32be8a'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:42.652497 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-d5618c97-9b50-4169-9481-bae1bc32be8a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7415bcf3ee6f426ba3f908b200ad3bb3', 'request_id': 'req-d5618c97-9b50-4169-9481-bae1bc32be8a'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:42.665651 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-d5618c97-9b50-4169-9481-bae1bc32be8a admin admin] The token cache is being invalidate because project 7415bcf3ee6f426ba3f908b200ad3bb3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81453) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 00:56:42.665939 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-d5618c97-9b50-4169-9481-bae1bc32be8a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:42.666556 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-d5618c97-9b50-4169-9481-bae1bc32be8a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:42.668066 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 884/1795] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 00:56:42 2022] DELETE /identity/v3/projects/7415bcf3ee6f426ba3f908b200ad3bb3 => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 09 00:56:42.672869 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2549465a-36e5-48aa-afb3-df1caff0f96f None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:42.673156 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2549465a-36e5-48aa-afb3-df1caff0f96f None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:42.673429 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2549465a-36e5-48aa-afb3-df1caff0f96f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:42.693236 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-2549465a-36e5-48aa-afb3-df1caff0f96f None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:56:42.694413 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-2549465a-36e5-48aa-afb3-df1caff0f96f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:42.753595 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 912/1796] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:56:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:56:42.762471 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d89ba81-a78f-4077-8744-f89b34ed3f7d None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:42.762760 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d89ba81-a78f-4077-8744-f89b34ed3f7d None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:42.763035 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d89ba81-a78f-4077-8744-f89b34ed3f7d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:42.787271 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-0d89ba81-a78f-4077-8744-f89b34ed3f7d None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:56:42.788616 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-0d89ba81-a78f-4077-8744-f89b34ed3f7d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:42.797190 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 885/1797] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:56:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:56:42.802077 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2549465a-36e5-48aa-afb3-df1caff0f96f None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:42.812764 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2549465a-36e5-48aa-afb3-df1caff0f96f None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:42.817895 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb65a898-6144-4fdf-9c7b-d1be221f34f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:42.818493 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb65a898-6144-4fdf-9c7b-d1be221f34f5 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:42.818799 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb65a898-6144-4fdf-9c7b-d1be221f34f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:42.819072 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb65a898-6144-4fdf-9c7b-d1be221f34f5 admin admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:42.819872 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb65a898-6144-4fdf-9c7b-d1be221f34f5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:56:42.820119 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb65a898-6144-4fdf-9c7b-d1be221f34f5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:42.821624 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb65a898-6144-4fdf-9c7b-d1be221f34f5 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:42.834409 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 913/1798] 10.222.0.33 () {58 vars in 1153 bytes} [Sun Oct 9 00:56:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:56:42.850263 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d89ba81-a78f-4077-8744-f89b34ed3f7d None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:42.859692 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d89ba81-a78f-4077-8744-f89b34ed3f7d None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:42.864450 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eea8f59-2913-48eb-979c-25e9ee75c1b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:42.865080 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eea8f59-2913-48eb-979c-25e9ee75c1b9 admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:42.865362 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eea8f59-2913-48eb-979c-25e9ee75c1b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:42.865637 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eea8f59-2913-48eb-979c-25e9ee75c1b9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:42.866277 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eea8f59-2913-48eb-979c-25e9ee75c1b9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:42.868217 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eea8f59-2913-48eb-979c-25e9ee75c1b9 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:42.897416 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 886/1799] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 00:56:42 2022] POST /identity/v3/projects => generated 393 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:56:42.901898 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb65a898-6144-4fdf-9c7b-d1be221f34f5 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:42.912666 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb65a898-6144-4fdf-9c7b-d1be221f34f5 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:42.954449 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:56:42.991077 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60a629c3-7905-4ef2-a1a6-d52f228358fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:42.991740 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60a629c3-7905-4ef2-a1a6-d52f228358fd admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:42.992022 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60a629c3-7905-4ef2-a1a6-d52f228358fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:42.992292 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60a629c3-7905-4ef2-a1a6-d52f228358fd admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:42.992932 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60a629c3-7905-4ef2-a1a6-d52f228358fd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:42.994885 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60a629c3-7905-4ef2-a1a6-d52f228358fd admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:43.017582 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 914/1800] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:56:42 2022] POST /identity/v3/users => generated 297 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:56:43.021867 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eea8f59-2913-48eb-979c-25e9ee75c1b9 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:43.033748 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eea8f59-2913-48eb-979c-25e9ee75c1b9 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:43.039797 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1602d78-fc74-44e2-8e23-aeb4fc72e5c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:43.040759 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1602d78-fc74-44e2-8e23-aeb4fc72e5c0 admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:43.041192 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1602d78-fc74-44e2-8e23-aeb4fc72e5c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:43.041625 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1602d78-fc74-44e2-8e23-aeb4fc72e5c0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:43.042461 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1602d78-fc74-44e2-8e23-aeb4fc72e5c0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:43.044278 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1602d78-fc74-44e2-8e23-aeb4fc72e5c0 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:43.057403 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 887/1801] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:56:43 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:56:43.059352 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60a629c3-7905-4ef2-a1a6-d52f228358fd admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:43.069996 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60a629c3-7905-4ef2-a1a6-d52f228358fd admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:43.074699 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f943b6a-8ffa-4662-b224-25c336e8ec75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:43.075561 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f943b6a-8ffa-4662-b224-25c336e8ec75 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:43.075894 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f943b6a-8ffa-4662-b224-25c336e8ec75 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:43.076173 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f943b6a-8ffa-4662-b224-25c336e8ec75 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:43.076817 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f943b6a-8ffa-4662-b224-25c336e8ec75 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:43.078417 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f943b6a-8ffa-4662-b224-25c336e8ec75 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:43.090736 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 915/1802] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:56:43 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:56:43.094961 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1602d78-fc74-44e2-8e23-aeb4fc72e5c0 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:43.105137 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1602d78-fc74-44e2-8e23-aeb4fc72e5c0 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:43.110056 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d8b9b8a-3dbb-42e8-9bbd-0338281c5aae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:43.110711 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d8b9b8a-3dbb-42e8-9bbd-0338281c5aae admin admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:43.110994 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d8b9b8a-3dbb-42e8-9bbd-0338281c5aae admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:43.111270 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d8b9b8a-3dbb-42e8-9bbd-0338281c5aae admin admin] PATH_INFO: `/v3/projects/f8a9c991802a4922af406d3734336bb5/users/38a567dc982e44c7912ca1fb5395b238/roles/588016933bab436d8b7c3ac76827356d` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:43.126199 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d8b9b8a-3dbb-42e8-9bbd-0338281c5aae admin admin] RBAC: Authorizing `identity:create_grant(project_id=f8a9c991802a4922af406d3734336bb5, user_id=38a567dc982e44c7912ca1fb5395b238, role_id=588016933bab436d8b7c3ac76827356d)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:43.129514 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d8b9b8a-3dbb-42e8-9bbd-0338281c5aae admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:43.142879 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 888/1803] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:56:43 2022] PUT /identity/v3/projects/f8a9c991802a4922af406d3734336bb5/users/38a567dc982e44c7912ca1fb5395b238/roles/588016933bab436d8b7c3ac76827356d => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:56:43.153837 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02158a78-36d7-4b90-b367-016bf8fcbaea None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:43.154203 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02158a78-36d7-4b90-b367-016bf8fcbaea None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:43.154540 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02158a78-36d7-4b90-b367-016bf8fcbaea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:43.182337 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-02158a78-36d7-4b90-b367-016bf8fcbaea None None] MFA Rules not processed for user `38a567dc982e44c7912ca1fb5395b238`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:56:43.183701 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-02158a78-36d7-4b90-b367-016bf8fcbaea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:43.259647 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 916/1804] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:56:43 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:56:43.264125 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d8b9b8a-3dbb-42e8-9bbd-0338281c5aae admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:43.326136 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d8b9b8a-3dbb-42e8-9bbd-0338281c5aae admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:43.331333 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4da6bb3b-c6de-405e-94dc-bcc2d87883dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:43.331965 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da6bb3b-c6de-405e-94dc-bcc2d87883dc admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:43.332249 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da6bb3b-c6de-405e-94dc-bcc2d87883dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:43.332522 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da6bb3b-c6de-405e-94dc-bcc2d87883dc admin admin] PATH_INFO: `/v3/projects` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:43.333234 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4da6bb3b-c6de-405e-94dc-bcc2d87883dc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:43.335144 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4da6bb3b-c6de-405e-94dc-bcc2d87883dc admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:43.362414 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 889/1805] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 00:56:43 2022] POST /identity/v3/projects => generated 393 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:56:43.366377 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02158a78-36d7-4b90-b367-016bf8fcbaea None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:43.377194 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02158a78-36d7-4b90-b367-016bf8fcbaea None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:43.435655 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8c74984-14ee-4f6c-8947-3fd8de366999 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:43.436266 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c74984-14ee-4f6c-8947-3fd8de366999 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:43.436547 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c74984-14ee-4f6c-8947-3fd8de366999 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:43.437319 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c74984-14ee-4f6c-8947-3fd8de366999 admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:43.437508 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8c74984-14ee-4f6c-8947-3fd8de366999 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:43.439425 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8c74984-14ee-4f6c-8947-3fd8de366999 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:43.461612 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 917/1806] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:56:43 2022] POST /identity/v3/users => generated 303 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:56:43.465596 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4da6bb3b-c6de-405e-94dc-bcc2d87883dc admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:43.476637 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4da6bb3b-c6de-405e-94dc-bcc2d87883dc admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:43.481238 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b56bfee-5055-4e07-b458-d4f489ecd460 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:43.481840 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b56bfee-5055-4e07-b458-d4f489ecd460 admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:43.482122 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b56bfee-5055-4e07-b458-d4f489ecd460 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:43.482399 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b56bfee-5055-4e07-b458-d4f489ecd460 admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:43.483032 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b56bfee-5055-4e07-b458-d4f489ecd460 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:43.484673 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b56bfee-5055-4e07-b458-d4f489ecd460 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:43.495005 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 890/1807] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:56:43 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:56:43.499071 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8c74984-14ee-4f6c-8947-3fd8de366999 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:43.508881 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8c74984-14ee-4f6c-8947-3fd8de366999 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:43.513321 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfb34d16-fcee-4d28-9b54-032e5169d0cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:43.513951 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb34d16-fcee-4d28-9b54-032e5169d0cf admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:43.514229 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb34d16-fcee-4d28-9b54-032e5169d0cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:43.514501 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb34d16-fcee-4d28-9b54-032e5169d0cf admin admin] PATH_INFO: `/v3/projects/0df83ee2d3544020aff2bb52b6aef9cb/users/07d86a2e3dfc4e95baa2a82a66a68a85/roles/092036ee46ac427baf4d56468575f697` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:43.529116 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfb34d16-fcee-4d28-9b54-032e5169d0cf admin admin] RBAC: Authorizing `identity:create_grant(project_id=0df83ee2d3544020aff2bb52b6aef9cb, user_id=07d86a2e3dfc4e95baa2a82a66a68a85, role_id=092036ee46ac427baf4d56468575f697)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:43.532033 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfb34d16-fcee-4d28-9b54-032e5169d0cf admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:43.545249 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 918/1808] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:56:43 2022] PUT /identity/v3/projects/0df83ee2d3544020aff2bb52b6aef9cb/users/07d86a2e3dfc4e95baa2a82a66a68a85/roles/092036ee46ac427baf4d56468575f697 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 09 00:56:43.549120 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b56bfee-5055-4e07-b458-d4f489ecd460 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:43.559045 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b56bfee-5055-4e07-b458-d4f489ecd460 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:43.578709 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:DELETE=6,keystone:SELECT=90,keystone:INSERT=4 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:56:43.615897 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d7e7cdf-64a7-47f1-a43f-9dad7dbe97d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:43.616515 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d7e7cdf-64a7-47f1-a43f-9dad7dbe97d3 admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:43.616787 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d7e7cdf-64a7-47f1-a43f-9dad7dbe97d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:43.617060 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d7e7cdf-64a7-47f1-a43f-9dad7dbe97d3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:43.617690 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d7e7cdf-64a7-47f1-a43f-9dad7dbe97d3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:43.619272 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d7e7cdf-64a7-47f1-a43f-9dad7dbe97d3 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:43.629675 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 891/1809] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:56:43 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 09 00:56:43.633926 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfb34d16-fcee-4d28-9b54-032e5169d0cf admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:43.639313 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:56:43.645946 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfb34d16-fcee-4d28-9b54-032e5169d0cf admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:43.650960 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a67f4847-672f-421b-83b6-352ee1650602 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:43.651655 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67f4847-672f-421b-83b6-352ee1650602 admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:43.651980 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67f4847-672f-421b-83b6-352ee1650602 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:43.652258 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67f4847-672f-421b-83b6-352ee1650602 admin admin] PATH_INFO: `/v3/projects/0df83ee2d3544020aff2bb52b6aef9cb/users/07d86a2e3dfc4e95baa2a82a66a68a85/roles/092036ee46ac427baf4d56468575f697` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:43.654177 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a67f4847-672f-421b-83b6-352ee1650602 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0df83ee2d3544020aff2bb52b6aef9cb, user_id=07d86a2e3dfc4e95baa2a82a66a68a85, role_id=092036ee46ac427baf4d56468575f697)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:43.657105 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a67f4847-672f-421b-83b6-352ee1650602 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:43.667161 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 919/1810] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:56:43 2022] PUT /identity/v3/projects/0df83ee2d3544020aff2bb52b6aef9cb/users/07d86a2e3dfc4e95baa2a82a66a68a85/roles/092036ee46ac427baf4d56468575f697 => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:56:43.677294 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf67e0c3-14cf-497c-a2c7-5475fe72ef06 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:43.677582 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf67e0c3-14cf-497c-a2c7-5475fe72ef06 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:43.677868 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf67e0c3-14cf-497c-a2c7-5475fe72ef06 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:43.707164 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-cf67e0c3-14cf-497c-a2c7-5475fe72ef06 None None] MFA Rules not processed for user `07d86a2e3dfc4e95baa2a82a66a68a85`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:56:43.708330 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-cf67e0c3-14cf-497c-a2c7-5475fe72ef06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:43.778097 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 892/1811] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:56:43 2022] POST /identity/v3/auth/tokens => generated 3337 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:56:43.789737 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a67f4847-672f-421b-83b6-352ee1650602 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:43.791545 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-a67f4847-672f-421b-83b6-352ee1650602 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:43.890005 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a67f4847-672f-421b-83b6-352ee1650602 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:43.895086 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc532f85-4885-4359-af18-f3a0c2053738 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3f48335668a4e8f934ac11ff66cd308', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:43.895818 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc532f85-4885-4359-af18-f3a0c2053738 service cinder] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:43.896174 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc532f85-4885-4359-af18-f3a0c2053738 service cinder] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:43.896552 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc532f85-4885-4359-af18-f3a0c2053738 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:43.908336 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc532f85-4885-4359-af18-f3a0c2053738 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:43.910196 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc532f85-4885-4359-af18-f3a0c2053738 service cinder] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:43.925256 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 920/1812] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:56:43 2022] GET /identity/v3/auth/tokens => generated 3337 bytes in 136 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:56:44.001346 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf67e0c3-14cf-497c-a2c7-5475fe72ef06 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:44.013161 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf67e0c3-14cf-497c-a2c7-5475fe72ef06 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:44.017534 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36b2e8a7-e489-402a-9310-9336a19006a5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3f48335668a4e8f934ac11ff66cd308', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:44.018105 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b2e8a7-e489-402a-9310-9336a19006a5 service cinder] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:44.019521 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b2e8a7-e489-402a-9310-9336a19006a5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:44.019521 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b2e8a7-e489-402a-9310-9336a19006a5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:44.109421 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36b2e8a7-e489-402a-9310-9336a19006a5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:44.110864 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36b2e8a7-e489-402a-9310-9336a19006a5 service cinder] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:44.121764 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 893/1813] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:56:44 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 121 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:56:52.720474 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cac7440-cdcf-4730-a7d1-081afca7b623 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:52.721014 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cac7440-cdcf-4730-a7d1-081afca7b623 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:52.721014 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cac7440-cdcf-4730-a7d1-081afca7b623 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:52.722776 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 921/1814] 10.222.0.33 () {54 vars in 921 bytes} [Sun Oct 9 00:56:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:56:52.731307 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15937cc9-914c-44ad-b950-e1fa69b70768 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:52.732113 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15937cc9-914c-44ad-b950-e1fa69b70768 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:52.732567 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15937cc9-914c-44ad-b950-e1fa69b70768 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:52.755961 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-15937cc9-914c-44ad-b950-e1fa69b70768 None None] MFA Rules not processed for user `e0a5d95969e74157bb2d2a57207e1651`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:56:52.757398 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-15937cc9-914c-44ad-b950-e1fa69b70768 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:52.821689 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:56:52.875142 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 894/1815] 10.222.0.33 () {58 vars in 1018 bytes} [Sun Oct 9 00:56:52 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:56:53.019994 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cac7440-cdcf-4730-a7d1-081afca7b623 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:53.021709 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-3cac7440-cdcf-4730-a7d1-081afca7b623 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:53.078397 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cac7440-cdcf-4730-a7d1-081afca7b623 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:53.092532 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bbe6fcb-7861-4aba-940a-3ea097442b7c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a5d95969e74157bb2d2a57207e1651', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:53.093275 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bbe6fcb-7861-4aba-940a-3ea097442b7c service nova] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:53.093554 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bbe6fcb-7861-4aba-940a-3ea097442b7c service nova] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:53.094032 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bbe6fcb-7861-4aba-940a-3ea097442b7c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:53.099535 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15937cc9-914c-44ad-b950-e1fa69b70768 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:53.101113 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-15937cc9-914c-44ad-b950-e1fa69b70768 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:53.132414 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bbe6fcb-7861-4aba-940a-3ea097442b7c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:53.134009 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bbe6fcb-7861-4aba-940a-3ea097442b7c service nova] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:53.162652 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 922/1816] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:56:53 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 143 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:56:53.243291 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15937cc9-914c-44ad-b950-e1fa69b70768 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:53.254995 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-903c819d-3b5c-4f73-b75d-9d825d8eaeb2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:53.257640 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-903c819d-3b5c-4f73-b75d-9d825d8eaeb2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:53.258299 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-903c819d-3b5c-4f73-b75d-9d825d8eaeb2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:53.259297 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-903c819d-3b5c-4f73-b75d-9d825d8eaeb2 admin admin] PATH_INFO: `/v3/users/e4848d52d9384834a7946936c029c456` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:53.260737 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-903c819d-3b5c-4f73-b75d-9d825d8eaeb2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e4848d52d9384834a7946936c029c456)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:53.264288 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-903c819d-3b5c-4f73-b75d-9d825d8eaeb2 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:53.288445 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bbe6fcb-7861-4aba-940a-3ea097442b7c service nova] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:53.305531 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bbe6fcb-7861-4aba-940a-3ea097442b7c service nova] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:53.319977 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15a1e107-090a-4792-85ba-9995ed354adf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:53.320708 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a1e107-090a-4792-85ba-9995ed354adf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:53.321094 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a1e107-090a-4792-85ba-9995ed354adf admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:53.321467 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a1e107-090a-4792-85ba-9995ed354adf admin admin] PATH_INFO: `/v3/users/38a567dc982e44c7912ca1fb5395b238` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:53.322934 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15a1e107-090a-4792-85ba-9995ed354adf admin admin] RBAC: Authorizing `identity:delete_user(user_id=38a567dc982e44c7912ca1fb5395b238)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:53.326020 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15a1e107-090a-4792-85ba-9995ed354adf admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:53.337649 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-903c819d-3b5c-4f73-b75d-9d825d8eaeb2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e4848d52d9384834a7946936c029c456'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:53.348811 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-903c819d-3b5c-4f73-b75d-9d825d8eaeb2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e4848d52d9384834a7946936c029c456'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:53.350043 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-903c819d-3b5c-4f73-b75d-9d825d8eaeb2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e4848d52d9384834a7946936c029c456'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:53.365105 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-903c819d-3b5c-4f73-b75d-9d825d8eaeb2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e4848d52d9384834a7946936c029c456'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:53.401325 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-903c819d-3b5c-4f73-b75d-9d825d8eaeb2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e4848d52d9384834a7946936c029c456'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:53.415215 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 895/1817] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 00:56:53 2022] DELETE /identity/v3/users/e4848d52d9384834a7946936c029c456 => generated 0 bytes in 313 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:56:53.419398 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa114c6-19d4-419b-bc14-fdbe4c663194 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:53.419743 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa114c6-19d4-419b-bc14-fdbe4c663194 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:53.420029 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa114c6-19d4-419b-bc14-fdbe4c663194 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:53.424264 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-15a1e107-090a-4792-85ba-9995ed354adf admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '38a567dc982e44c7912ca1fb5395b238'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:53.460753 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-15a1e107-090a-4792-85ba-9995ed354adf admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '38a567dc982e44c7912ca1fb5395b238'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:53.462001 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-15a1e107-090a-4792-85ba-9995ed354adf admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '38a567dc982e44c7912ca1fb5395b238'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:53.462420 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-15a1e107-090a-4792-85ba-9995ed354adf admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '38a567dc982e44c7912ca1fb5395b238'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:53.469201 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-6fa114c6-19d4-419b-bc14-fdbe4c663194 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:56:53.470933 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-6fa114c6-19d4-419b-bc14-fdbe4c663194 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:53.482649 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-15a1e107-090a-4792-85ba-9995ed354adf admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '38a567dc982e44c7912ca1fb5395b238'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:53.535517 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 923/1818] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 00:56:53 2022] DELETE /identity/v3/users/38a567dc982e44c7912ca1fb5395b238 => generated 0 bytes in 252 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:56:53.538301 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c174235b-68f8-46d2-88de-23f30058428c None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:53.538822 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c174235b-68f8-46d2-88de-23f30058428c None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:53.539186 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c174235b-68f8-46d2-88de-23f30058428c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:53.556474 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:56:53.560272 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-c174235b-68f8-46d2-88de-23f30058428c None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:56:53.561562 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-c174235b-68f8-46d2-88de-23f30058428c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:53.579830 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 896/1819] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:56:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:56:53.593678 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fa114c6-19d4-419b-bc14-fdbe4c663194 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:53.605069 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-6fa114c6-19d4-419b-bc14-fdbe4c663194 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:53.628232 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:56:53.652404 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 924/1820] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:56:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:56:53.664424 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c174235b-68f8-46d2-88de-23f30058428c None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:53.666910 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-c174235b-68f8-46d2-88de-23f30058428c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:53.699998 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fa114c6-19d4-419b-bc14-fdbe4c663194 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:53.751618 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c174235b-68f8-46d2-88de-23f30058428c None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:53.756647 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-082d9650-eddc-40db-8426-84944f8b1c1a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:53.758338 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-082d9650-eddc-40db-8426-84944f8b1c1a service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:53.759164 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-082d9650-eddc-40db-8426-84944f8b1c1a service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:53.759817 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-082d9650-eddc-40db-8426-84944f8b1c1a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:53.772868 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-082d9650-eddc-40db-8426-84944f8b1c1a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:53.774967 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-082d9650-eddc-40db-8426-84944f8b1c1a service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:53.788547 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce9b0042-4061-4f6e-92fe-caed4d894052 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:53.789185 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce9b0042-4061-4f6e-92fe-caed4d894052 service neutron] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:53.789462 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce9b0042-4061-4f6e-92fe-caed4d894052 service neutron] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:53.789742 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce9b0042-4061-4f6e-92fe-caed4d894052 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:53.790286 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 925/1821] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:56:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 127 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:56:53.818244 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce9b0042-4061-4f6e-92fe-caed4d894052 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:53.819969 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce9b0042-4061-4f6e-92fe-caed4d894052 service neutron] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:53.830354 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 897/1822] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:56:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 238 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:56:54.341429 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-082d9650-eddc-40db-8426-84944f8b1c1a service neutron] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:54.343848 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-082d9650-eddc-40db-8426-84944f8b1c1a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:54.367083 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-082d9650-eddc-40db-8426-84944f8b1c1a service neutron] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:54.371995 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54679157-01cd-4991-bb36-b90cc78e0253 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:54.372720 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54679157-01cd-4991-bb36-b90cc78e0253 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:54.373242 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54679157-01cd-4991-bb36-b90cc78e0253 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:54.373726 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54679157-01cd-4991-bb36-b90cc78e0253 admin admin] PATH_INFO: `/v3/projects/f8a9c991802a4922af406d3734336bb5` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:54.375568 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54679157-01cd-4991-bb36-b90cc78e0253 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f8a9c991802a4922af406d3734336bb5)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:54.377623 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54679157-01cd-4991-bb36-b90cc78e0253 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:54.443432 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-54679157-01cd-4991-bb36-b90cc78e0253 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f8a9c991802a4922af406d3734336bb5', 'request_id': 'req-54679157-01cd-4991-bb36-b90cc78e0253'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:54.452533 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-54679157-01cd-4991-bb36-b90cc78e0253 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f8a9c991802a4922af406d3734336bb5', 'request_id': 'req-54679157-01cd-4991-bb36-b90cc78e0253'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:54.464424 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-54679157-01cd-4991-bb36-b90cc78e0253 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f8a9c991802a4922af406d3734336bb5', 'request_id': 'req-54679157-01cd-4991-bb36-b90cc78e0253'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:54.485237 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce9b0042-4061-4f6e-92fe-caed4d894052 service neutron] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:54.486971 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-ce9b0042-4061-4f6e-92fe-caed4d894052 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:54.501159 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-54679157-01cd-4991-bb36-b90cc78e0253 admin admin] The token cache is being invalidate because project f8a9c991802a4922af406d3734336bb5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 00:56:54.501322 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-54679157-01cd-4991-bb36-b90cc78e0253 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:54.502358 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-54679157-01cd-4991-bb36-b90cc78e0253 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:54.506506 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 926/1823] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 00:56:54 2022] DELETE /identity/v3/projects/f8a9c991802a4922af406d3734336bb5 => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:56:54.506649 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce9b0042-4061-4f6e-92fe-caed4d894052 service neutron] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:54.507791 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54679157-01cd-4991-bb36-b90cc78e0253 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:54.509648 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-54679157-01cd-4991-bb36-b90cc78e0253 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:54.599510 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98429497-2939-41d3-9d1d-0ed4e2e373a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:54.600193 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98429497-2939-41d3-9d1d-0ed4e2e373a0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:54.600477 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98429497-2939-41d3-9d1d-0ed4e2e373a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:54.600758 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98429497-2939-41d3-9d1d-0ed4e2e373a0 admin admin] PATH_INFO: `/v3/projects/2912461a18214cb5a7b0762205e64b1e` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:54.602002 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98429497-2939-41d3-9d1d-0ed4e2e373a0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2912461a18214cb5a7b0762205e64b1e)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:54.603961 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98429497-2939-41d3-9d1d-0ed4e2e373a0 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:54.617637 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54679157-01cd-4991-bb36-b90cc78e0253 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:54.623385 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e55ce3fa-d2ed-4e50-8b7e-d79714b94153 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:54.624121 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e55ce3fa-d2ed-4e50-8b7e-d79714b94153 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:54.624402 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e55ce3fa-d2ed-4e50-8b7e-d79714b94153 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:54.624671 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e55ce3fa-d2ed-4e50-8b7e-d79714b94153 admin admin] PATH_INFO: `/v3/users/07d86a2e3dfc4e95baa2a82a66a68a85` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:54.628726 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e55ce3fa-d2ed-4e50-8b7e-d79714b94153 admin admin] RBAC: Authorizing `identity:delete_user(user_id=07d86a2e3dfc4e95baa2a82a66a68a85)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:54.628726 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e55ce3fa-d2ed-4e50-8b7e-d79714b94153 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:54.654854 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:56:54.675857 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-98429497-2939-41d3-9d1d-0ed4e2e373a0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2912461a18214cb5a7b0762205e64b1e', 'request_id': 'req-98429497-2939-41d3-9d1d-0ed4e2e373a0'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:54.682707 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=94,keystone:DELETE=6 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:56:54.683983 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-98429497-2939-41d3-9d1d-0ed4e2e373a0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2912461a18214cb5a7b0762205e64b1e', 'request_id': 'req-98429497-2939-41d3-9d1d-0ed4e2e373a0'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:54.694323 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-98429497-2939-41d3-9d1d-0ed4e2e373a0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2912461a18214cb5a7b0762205e64b1e', 'request_id': 'req-98429497-2939-41d3-9d1d-0ed4e2e373a0'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:54.703163 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-e55ce3fa-d2ed-4e50-8b7e-d79714b94153 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '07d86a2e3dfc4e95baa2a82a66a68a85'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:54.715162 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-e55ce3fa-d2ed-4e50-8b7e-d79714b94153 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '07d86a2e3dfc4e95baa2a82a66a68a85'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:54.716243 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-e55ce3fa-d2ed-4e50-8b7e-d79714b94153 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '07d86a2e3dfc4e95baa2a82a66a68a85'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:54.717095 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-98429497-2939-41d3-9d1d-0ed4e2e373a0 admin admin] The token cache is being invalidate because project 2912461a18214cb5a7b0762205e64b1e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81453) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 00:56:54.717390 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-98429497-2939-41d3-9d1d-0ed4e2e373a0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:54.718149 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-98429497-2939-41d3-9d1d-0ed4e2e373a0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:54.718612 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-e55ce3fa-d2ed-4e50-8b7e-d79714b94153 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '07d86a2e3dfc4e95baa2a82a66a68a85'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:54.720106 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 898/1824] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 00:56:54 2022] DELETE /identity/v3/projects/2912461a18214cb5a7b0762205e64b1e => generated 0 bytes in 236 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 09 00:56:54.725692 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98429497-2939-41d3-9d1d-0ed4e2e373a0 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:54.727443 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-98429497-2939-41d3-9d1d-0ed4e2e373a0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:54.731566 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-e55ce3fa-d2ed-4e50-8b7e-d79714b94153 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '07d86a2e3dfc4e95baa2a82a66a68a85'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:54.766431 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 927/1825] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 00:56:54 2022] DELETE /identity/v3/users/07d86a2e3dfc4e95baa2a82a66a68a85 => generated 0 bytes in 258 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:56:54.826450 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98429497-2939-41d3-9d1d-0ed4e2e373a0 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:54.910199 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-485f824f-500e-452c-8510-dfc680107d19 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:54.910845 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485f824f-500e-452c-8510-dfc680107d19 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:54.911129 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485f824f-500e-452c-8510-dfc680107d19 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:54.911408 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485f824f-500e-452c-8510-dfc680107d19 admin admin] PATH_INFO: `/v3/users/604ccc8529494def8a7dc54747a1fd79` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:54.912636 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-485f824f-500e-452c-8510-dfc680107d19 admin admin] RBAC: Authorizing `identity:delete_user(user_id=604ccc8529494def8a7dc54747a1fd79)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:54.914678 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-485f824f-500e-452c-8510-dfc680107d19 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:54.975379 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-485f824f-500e-452c-8510-dfc680107d19 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '604ccc8529494def8a7dc54747a1fd79'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:54.982858 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-485f824f-500e-452c-8510-dfc680107d19 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '604ccc8529494def8a7dc54747a1fd79'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:54.984070 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-485f824f-500e-452c-8510-dfc680107d19 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '604ccc8529494def8a7dc54747a1fd79'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:54.992843 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-485f824f-500e-452c-8510-dfc680107d19 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '604ccc8529494def8a7dc54747a1fd79'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:55.012101 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-485f824f-500e-452c-8510-dfc680107d19 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '604ccc8529494def8a7dc54747a1fd79'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:55.020602 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 899/1826] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 00:56:54 2022] DELETE /identity/v3/users/604ccc8529494def8a7dc54747a1fd79 => generated 0 bytes in 296 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 09 00:56:55.077058 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e55ce3fa-d2ed-4e50-8b7e-d79714b94153 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:55.080658 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-e55ce3fa-d2ed-4e50-8b7e-d79714b94153 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:55.100573 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e55ce3fa-d2ed-4e50-8b7e-d79714b94153 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:55.164562 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c7cfeb-5a5a-42b7-841e-3d8c0850bde1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:55.165217 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c7cfeb-5a5a-42b7-841e-3d8c0850bde1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:55.165498 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c7cfeb-5a5a-42b7-841e-3d8c0850bde1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:55.165773 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c7cfeb-5a5a-42b7-841e-3d8c0850bde1 admin admin] PATH_INFO: `/v3/users/9485c032a2d5462486ac73373db67a10` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:55.166911 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56c7cfeb-5a5a-42b7-841e-3d8c0850bde1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9485c032a2d5462486ac73373db67a10)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:55.168883 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56c7cfeb-5a5a-42b7-841e-3d8c0850bde1 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:55.223876 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-56c7cfeb-5a5a-42b7-841e-3d8c0850bde1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9485c032a2d5462486ac73373db67a10'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:55.233929 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-56c7cfeb-5a5a-42b7-841e-3d8c0850bde1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9485c032a2d5462486ac73373db67a10'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:55.234943 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-56c7cfeb-5a5a-42b7-841e-3d8c0850bde1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9485c032a2d5462486ac73373db67a10'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:55.235637 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-56c7cfeb-5a5a-42b7-841e-3d8c0850bde1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9485c032a2d5462486ac73373db67a10'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:55.247227 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-56c7cfeb-5a5a-42b7-841e-3d8c0850bde1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9485c032a2d5462486ac73373db67a10'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:55.277247 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 928/1827] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 00:56:55 2022] DELETE /identity/v3/users/9485c032a2d5462486ac73373db67a10 => generated 0 bytes in 197 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:56:55.278906 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5982c1a-00de-40c7-839f-ac3e69434f70 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:55.279200 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5982c1a-00de-40c7-839f-ac3e69434f70 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:55.279483 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5982c1a-00de-40c7-839f-ac3e69434f70 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:55.310268 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-c5982c1a-00de-40c7-839f-ac3e69434f70 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:56:55.311496 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-c5982c1a-00de-40c7-839f-ac3e69434f70 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:55.321702 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=87,keystone:INSERT=2,keystone:DELETE=11 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:56:55.322572 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c7cfeb-5a5a-42b7-841e-3d8c0850bde1 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:55.324393 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-56c7cfeb-5a5a-42b7-841e-3d8c0850bde1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:55.375693 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:DELETE=18,keystone:SELECT=80,keystone:INSERT=2 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:56:55.435504 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 900/1828] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:56:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:56:55.452048 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5982c1a-00de-40c7-839f-ac3e69434f70 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:55.458463 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-c5982c1a-00de-40c7-839f-ac3e69434f70 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:55.500137 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c7cfeb-5a5a-42b7-841e-3d8c0850bde1 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:55.508258 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bd0c7e1-bfa9-4c11-acc3-21f7ac105176 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:55.509662 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bd0c7e1-bfa9-4c11-acc3-21f7ac105176 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:55.510038 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bd0c7e1-bfa9-4c11-acc3-21f7ac105176 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:55.510480 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bd0c7e1-bfa9-4c11-acc3-21f7ac105176 admin admin] PATH_INFO: `/v3/projects/0df83ee2d3544020aff2bb52b6aef9cb` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:55.512850 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bd0c7e1-bfa9-4c11-acc3-21f7ac105176 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0df83ee2d3544020aff2bb52b6aef9cb)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:55.516032 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bd0c7e1-bfa9-4c11-acc3-21f7ac105176 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:55.589397 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-5bd0c7e1-bfa9-4c11-acc3-21f7ac105176 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0df83ee2d3544020aff2bb52b6aef9cb', 'request_id': 'req-5bd0c7e1-bfa9-4c11-acc3-21f7ac105176'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:55.600785 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-5bd0c7e1-bfa9-4c11-acc3-21f7ac105176 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0df83ee2d3544020aff2bb52b6aef9cb', 'request_id': 'req-5bd0c7e1-bfa9-4c11-acc3-21f7ac105176'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:55.609785 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-5bd0c7e1-bfa9-4c11-acc3-21f7ac105176 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0df83ee2d3544020aff2bb52b6aef9cb', 'request_id': 'req-5bd0c7e1-bfa9-4c11-acc3-21f7ac105176'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:55.621894 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5982c1a-00de-40c7-839f-ac3e69434f70 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:55.658427 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-5bd0c7e1-bfa9-4c11-acc3-21f7ac105176 admin admin] The token cache is being invalidate because project 0df83ee2d3544020aff2bb52b6aef9cb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 00:56:55.659266 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-5bd0c7e1-bfa9-4c11-acc3-21f7ac105176 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:55.660592 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-5bd0c7e1-bfa9-4c11-acc3-21f7ac105176 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:55.663332 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 929/1829] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 00:56:55 2022] DELETE /identity/v3/projects/0df83ee2d3544020aff2bb52b6aef9cb => generated 0 bytes in 342 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:56:55.668222 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-756e9c29-d989-41cd-9fac-68a753067aec None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:55.669105 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-756e9c29-d989-41cd-9fac-68a753067aec None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:55.669626 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-756e9c29-d989-41cd-9fac-68a753067aec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:55.694727 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-756e9c29-d989-41cd-9fac-68a753067aec None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:56:55.696095 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-756e9c29-d989-41cd-9fac-68a753067aec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:55.733242 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0443021-2eb6-4cf5-a245-076805b7b8d1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:55.734013 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0443021-2eb6-4cf5-a245-076805b7b8d1 service neutron] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:55.734529 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0443021-2eb6-4cf5-a245-076805b7b8d1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:55.735014 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0443021-2eb6-4cf5-a245-076805b7b8d1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:55.739461 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-a0443021-2eb6-4cf5-a245-076805b7b8d1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:55.789103 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 930/1830] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:56:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 09 00:56:55.789940 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-756e9c29-d989-41cd-9fac-68a753067aec None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:55.791204 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-756e9c29-d989-41cd-9fac-68a753067aec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:55.803774 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-756e9c29-d989-41cd-9fac-68a753067aec None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:55.808492 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-742cf461-aa0e-41b4-835e-5451e84e4246 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:55.809098 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-742cf461-aa0e-41b4-835e-5451e84e4246 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:55.809370 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-742cf461-aa0e-41b4-835e-5451e84e4246 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:55.809643 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-742cf461-aa0e-41b4-835e-5451e84e4246 admin admin] PATH_INFO: `/v3/projects/abf32db91d464fe4b470eebe5bf0c78d` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:55.810733 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-742cf461-aa0e-41b4-835e-5451e84e4246 admin admin] RBAC: Authorizing `identity:delete_project(project_id=abf32db91d464fe4b470eebe5bf0c78d)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:55.812685 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-742cf461-aa0e-41b4-835e-5451e84e4246 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:55.826380 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0443021-2eb6-4cf5-a245-076805b7b8d1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:55.827837 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0443021-2eb6-4cf5-a245-076805b7b8d1 service neutron] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:55.839907 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 901/1831] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:56:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 389 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 09 00:56:55.841247 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a6c5d7-10e2-4aa4-a7f5-16361e527354 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:55.841653 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a6c5d7-10e2-4aa4-a7f5-16361e527354 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:55.841953 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a6c5d7-10e2-4aa4-a7f5-16361e527354 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:55.858610 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:56:55.866000 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-31a6c5d7-10e2-4aa4-a7f5-16361e527354 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:56:55.867223 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-31a6c5d7-10e2-4aa4-a7f5-16361e527354 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:55.867575 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-742cf461-aa0e-41b4-835e-5451e84e4246 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'abf32db91d464fe4b470eebe5bf0c78d', 'request_id': 'req-742cf461-aa0e-41b4-835e-5451e84e4246'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:55.876236 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-742cf461-aa0e-41b4-835e-5451e84e4246 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'abf32db91d464fe4b470eebe5bf0c78d', 'request_id': 'req-742cf461-aa0e-41b4-835e-5451e84e4246'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:55.881664 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:56:55.883638 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-742cf461-aa0e-41b4-835e-5451e84e4246 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'abf32db91d464fe4b470eebe5bf0c78d', 'request_id': 'req-742cf461-aa0e-41b4-835e-5451e84e4246'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:55.901225 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-742cf461-aa0e-41b4-835e-5451e84e4246 admin admin] The token cache is being invalidate because project abf32db91d464fe4b470eebe5bf0c78d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 00:56:55.901566 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-742cf461-aa0e-41b4-835e-5451e84e4246 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:55.902255 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-742cf461-aa0e-41b4-835e-5451e84e4246 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:55.903923 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 931/1832] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 00:56:55 2022] DELETE /identity/v3/projects/abf32db91d464fe4b470eebe5bf0c78d => generated 0 bytes in 114 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:56:55.909997 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1334be8-17f8-4982-9183-380bf71159f0 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:55.910493 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1334be8-17f8-4982-9183-380bf71159f0 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:55.910614 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1334be8-17f8-4982-9183-380bf71159f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:55.943724 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-d1334be8-17f8-4982-9183-380bf71159f0 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:56:55.945167 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-d1334be8-17f8-4982-9183-380bf71159f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:55.973876 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 902/1833] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:56:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:56:55.975630 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31a6c5d7-10e2-4aa4-a7f5-16361e527354 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:55.986096 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31a6c5d7-10e2-4aa4-a7f5-16361e527354 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:56.008731 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82a08bcb-6536-4413-87a5-e618e07987e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:56.009562 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a08bcb-6536-4413-87a5-e618e07987e8 admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:56.009853 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a08bcb-6536-4413-87a5-e618e07987e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:56.010133 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a08bcb-6536-4413-87a5-e618e07987e8 admin admin] PATH_INFO: `/v3/domains` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:56.010789 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82a08bcb-6536-4413-87a5-e618e07987e8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:56:56.011111 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82a08bcb-6536-4413-87a5-e618e07987e8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:56.014462 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82a08bcb-6536-4413-87a5-e618e07987e8 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:56.039433 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 903/1834] 10.222.0.33 () {58 vars in 1153 bytes} [Sun Oct 9 00:56:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 65 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:56:56.044113 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82a08bcb-6536-4413-87a5-e618e07987e8 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:56.055382 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82a08bcb-6536-4413-87a5-e618e07987e8 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:56.062283 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0e67039-95b6-4cb8-b6a3-fb07b3fd4178 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:56.062902 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e67039-95b6-4cb8-b6a3-fb07b3fd4178 admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:56.063561 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e67039-95b6-4cb8-b6a3-fb07b3fd4178 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:56.063561 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e67039-95b6-4cb8-b6a3-fb07b3fd4178 admin admin] PATH_INFO: `/v3/projects` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:56.064326 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0e67039-95b6-4cb8-b6a3-fb07b3fd4178 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:56.066479 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0e67039-95b6-4cb8-b6a3-fb07b3fd4178 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:56.087322 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 932/1835] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:56:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 09 00:56:56.096851 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4ee154-2639-40c4-b7d7-2675ed539dbc None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:56.097277 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4ee154-2639-40c4-b7d7-2675ed539dbc None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:56.097789 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4ee154-2639-40c4-b7d7-2675ed539dbc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:56.108089 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 904/1836] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 00:56:56 2022] POST /identity/v3/projects => generated 407 bytes in 61 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:56:56.109268 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0e67039-95b6-4cb8-b6a3-fb07b3fd4178 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:56.146120 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-fd4ee154-2639-40c4-b7d7-2675ed539dbc None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:56:56.147463 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-fd4ee154-2639-40c4-b7d7-2675ed539dbc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:56.200554 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0e67039-95b6-4cb8-b6a3-fb07b3fd4178 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:56.208702 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ece75ae9-6d9e-4e8c-99dc-a8ab5e862937 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:56.209711 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece75ae9-6d9e-4e8c-99dc-a8ab5e862937 admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:56.209996 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece75ae9-6d9e-4e8c-99dc-a8ab5e862937 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:56.210265 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece75ae9-6d9e-4e8c-99dc-a8ab5e862937 admin admin] PATH_INFO: `/v3/users` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:56.211266 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ece75ae9-6d9e-4e8c-99dc-a8ab5e862937 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:56.214329 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ece75ae9-6d9e-4e8c-99dc-a8ab5e862937 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:56.235364 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 933/1837] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:56:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 09 00:56:56.239885 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd4ee154-2639-40c4-b7d7-2675ed539dbc None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:56.243752 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 905/1838] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:56:56 2022] POST /identity/v3/users => generated 304 bytes in 135 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:56:56.250761 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd4ee154-2639-40c4-b7d7-2675ed539dbc None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:56.253711 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ece75ae9-6d9e-4e8c-99dc-a8ab5e862937 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:56.255846 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81c1a9d5-a47f-4c2b-bd8d-a491864fb33c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:56.257092 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c1a9d5-a47f-4c2b-bd8d-a491864fb33c admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:56.257092 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c1a9d5-a47f-4c2b-bd8d-a491864fb33c admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:56.257092 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c1a9d5-a47f-4c2b-bd8d-a491864fb33c admin admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:56.257744 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81c1a9d5-a47f-4c2b-bd8d-a491864fb33c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:56:56.258053 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81c1a9d5-a47f-4c2b-bd8d-a491864fb33c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:56.259885 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81c1a9d5-a47f-4c2b-bd8d-a491864fb33c admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:56.273430 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ece75ae9-6d9e-4e8c-99dc-a8ab5e862937 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:56.278179 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 934/1839] 10.222.0.33 () {58 vars in 1153 bytes} [Sun Oct 9 00:56:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:56:56.284842 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81c1a9d5-a47f-4c2b-bd8d-a491864fb33c admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:56.290265 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-000ee93e-b5c8-4d0f-90f3-c947d496876a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:56.290901 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-000ee93e-b5c8-4d0f-90f3-c947d496876a admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:56.291717 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-000ee93e-b5c8-4d0f-90f3-c947d496876a admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:56.292005 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-000ee93e-b5c8-4d0f-90f3-c947d496876a admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:56.293330 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-000ee93e-b5c8-4d0f-90f3-c947d496876a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:56.295269 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-000ee93e-b5c8-4d0f-90f3-c947d496876a admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:56.296946 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81c1a9d5-a47f-4c2b-bd8d-a491864fb33c admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:56.302809 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08d25da1-3e78-417f-9346-9c71efc0c129 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:56.304288 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d25da1-3e78-417f-9346-9c71efc0c129 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:56.304587 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d25da1-3e78-417f-9346-9c71efc0c129 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:56.304866 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d25da1-3e78-417f-9346-9c71efc0c129 admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:56.305507 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08d25da1-3e78-417f-9346-9c71efc0c129 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:56.306182 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 906/1840] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:56:56 2022] GET /identity/v3/roles => generated 1420 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:56:56.307524 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08d25da1-3e78-417f-9346-9c71efc0c129 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:56.310116 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-000ee93e-b5c8-4d0f-90f3-c947d496876a admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:56.321310 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-000ee93e-b5c8-4d0f-90f3-c947d496876a admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:56.326192 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bb0dac4-974e-413e-95b7-351444c37e3c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:56.326807 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bb0dac4-974e-413e-95b7-351444c37e3c admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:56.327092 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bb0dac4-974e-413e-95b7-351444c37e3c admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:56.327370 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bb0dac4-974e-413e-95b7-351444c37e3c admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:56.328395 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bb0dac4-974e-413e-95b7-351444c37e3c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:56.330168 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bb0dac4-974e-413e-95b7-351444c37e3c admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:56.347569 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 907/1841] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:56:56 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 09 00:56:56.349335 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bb0dac4-974e-413e-95b7-351444c37e3c admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:56.351787 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-6bb0dac4-974e-413e-95b7-351444c37e3c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:56.366403 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 935/1842] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 00:56:56 2022] POST /identity/v3/projects => generated 387 bytes in 84 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 09 00:56:56.366664 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bb0dac4-974e-413e-95b7-351444c37e3c admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:56.367570 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08d25da1-3e78-417f-9346-9c71efc0c129 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:56.421226 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:56:56.435964 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8edc1cff-1008-4f42-8772-23e04f67d1d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:56.436617 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8edc1cff-1008-4f42-8772-23e04f67d1d4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:56.436749 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=94,keystone:DELETE=4,keystone:INSERT=2 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:56:56.436935 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8edc1cff-1008-4f42-8772-23e04f67d1d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:56.437165 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8edc1cff-1008-4f42-8772-23e04f67d1d4 admin admin] PATH_INFO: `/v3/projects/b129a63b5ece4a578c2bcc961efe5060` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:56.438322 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8edc1cff-1008-4f42-8772-23e04f67d1d4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b129a63b5ece4a578c2bcc961efe5060)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:56.441023 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8edc1cff-1008-4f42-8772-23e04f67d1d4 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:56.467414 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08d25da1-3e78-417f-9346-9c71efc0c129 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:56.473958 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1381d36d-e7cf-4e11-99a4-fc2ef3b0ca04 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:56.474912 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1381d36d-e7cf-4e11-99a4-fc2ef3b0ca04 admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:56.475402 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1381d36d-e7cf-4e11-99a4-fc2ef3b0ca04 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:56.476147 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1381d36d-e7cf-4e11-99a4-fc2ef3b0ca04 admin admin] PATH_INFO: `/v3/projects/5260f11ec8f645cba5394181e4d191cf/users/1fd3830f6a054cbabb3ca141f2a684f7/roles/588016933bab436d8b7c3ac76827356d` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:56.491884 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1381d36d-e7cf-4e11-99a4-fc2ef3b0ca04 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5260f11ec8f645cba5394181e4d191cf, user_id=1fd3830f6a054cbabb3ca141f2a684f7, role_id=588016933bab436d8b7c3ac76827356d)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:56.494668 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1381d36d-e7cf-4e11-99a4-fc2ef3b0ca04 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:56.498217 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-8edc1cff-1008-4f42-8772-23e04f67d1d4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b129a63b5ece4a578c2bcc961efe5060', 'request_id': 'req-8edc1cff-1008-4f42-8772-23e04f67d1d4'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:56.505373 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-8edc1cff-1008-4f42-8772-23e04f67d1d4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b129a63b5ece4a578c2bcc961efe5060', 'request_id': 'req-8edc1cff-1008-4f42-8772-23e04f67d1d4'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:56.512332 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 936/1843] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:56:56 2022] PUT /identity/v3/projects/5260f11ec8f645cba5394181e4d191cf/users/1fd3830f6a054cbabb3ca141f2a684f7/roles/588016933bab436d8b7c3ac76827356d => generated 0 bytes in 145 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:56:56.513350 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1381d36d-e7cf-4e11-99a4-fc2ef3b0ca04 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:56.516302 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-8edc1cff-1008-4f42-8772-23e04f67d1d4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b129a63b5ece4a578c2bcc961efe5060', 'request_id': 'req-8edc1cff-1008-4f42-8772-23e04f67d1d4'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:56.532592 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-8edc1cff-1008-4f42-8772-23e04f67d1d4 admin admin] The token cache is being invalidate because project b129a63b5ece4a578c2bcc961efe5060 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81453) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 00:56:56.532876 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-8edc1cff-1008-4f42-8772-23e04f67d1d4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:56.533521 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-8edc1cff-1008-4f42-8772-23e04f67d1d4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:56.538189 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 908/1844] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 00:56:56 2022] DELETE /identity/v3/projects/b129a63b5ece4a578c2bcc961efe5060 => generated 0 bytes in 187 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:56:56.538189 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec8a64c-b91a-480e-bc75-46d8b741f377 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:56.538189 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec8a64c-b91a-480e-bc75-46d8b741f377 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:56.538189 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec8a64c-b91a-480e-bc75-46d8b741f377 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:56.570354 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-5ec8a64c-b91a-480e-bc75-46d8b741f377 None None] MFA Rules not processed for user `1fd3830f6a054cbabb3ca141f2a684f7`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:56:56.571710 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-5ec8a64c-b91a-480e-bc75-46d8b741f377 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:56.602956 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1381d36d-e7cf-4e11-99a4-fc2ef3b0ca04 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:56.605413 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dc772d2-32c9-4166-ae49-7c9762a0325c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:56.606015 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dc772d2-32c9-4166-ae49-7c9762a0325c admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:56.606289 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dc772d2-32c9-4166-ae49-7c9762a0325c admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:56.606556 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dc772d2-32c9-4166-ae49-7c9762a0325c admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:56.607179 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dc772d2-32c9-4166-ae49-7c9762a0325c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:56.609315 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dc772d2-32c9-4166-ae49-7c9762a0325c admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:56.632292 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 937/1845] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:56:56 2022] POST /identity/v3/users => generated 294 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:56:56.633191 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dc772d2-32c9-4166-ae49-7c9762a0325c admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:56.634785 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-6dc772d2-32c9-4166-ae49-7c9762a0325c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:56.644441 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 909/1846] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:56:56 2022] POST /identity/v3/auth/tokens => generated 3284 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:56:56.647809 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ec8a64c-b91a-480e-bc75-46d8b741f377 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:56.647809 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-5ec8a64c-b91a-480e-bc75-46d8b741f377 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:56.647966 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dc772d2-32c9-4166-ae49-7c9762a0325c admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:56.652307 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53850d3b-99dd-4f37-82b8-88ce642c4bf8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:56.652931 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53850d3b-99dd-4f37-82b8-88ce642c4bf8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:56.653212 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53850d3b-99dd-4f37-82b8-88ce642c4bf8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:56.653487 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53850d3b-99dd-4f37-82b8-88ce642c4bf8 admin admin] PATH_INFO: `/v3/users/098962dfac444426a5d60f7fca1f3efd` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:56.654611 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53850d3b-99dd-4f37-82b8-88ce642c4bf8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=098962dfac444426a5d60f7fca1f3efd)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:56.656856 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53850d3b-99dd-4f37-82b8-88ce642c4bf8 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:56.660215 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ec8a64c-b91a-480e-bc75-46d8b741f377 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:56.664721 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-628bfc87-743f-4520-a3f7-9379addb2585 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:56.665323 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-628bfc87-743f-4520-a3f7-9379addb2585 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:56.665602 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-628bfc87-743f-4520-a3f7-9379addb2585 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:56.665871 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-628bfc87-743f-4520-a3f7-9379addb2585 admin admin] PATH_INFO: `/v3/users/f4d29c290aa048f591948a8915e802fd` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:56.666972 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-628bfc87-743f-4520-a3f7-9379addb2585 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f4d29c290aa048f591948a8915e802fd)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:56.668879 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-628bfc87-743f-4520-a3f7-9379addb2585 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:56.705107 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-53850d3b-99dd-4f37-82b8-88ce642c4bf8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '098962dfac444426a5d60f7fca1f3efd'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:56.712632 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-53850d3b-99dd-4f37-82b8-88ce642c4bf8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '098962dfac444426a5d60f7fca1f3efd'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:56.713331 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-53850d3b-99dd-4f37-82b8-88ce642c4bf8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '098962dfac444426a5d60f7fca1f3efd'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:56.714108 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-53850d3b-99dd-4f37-82b8-88ce642c4bf8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '098962dfac444426a5d60f7fca1f3efd'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:56.716600 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-628bfc87-743f-4520-a3f7-9379addb2585 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f4d29c290aa048f591948a8915e802fd'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:56.721715 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-53850d3b-99dd-4f37-82b8-88ce642c4bf8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '098962dfac444426a5d60f7fca1f3efd'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:56.724819 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-628bfc87-743f-4520-a3f7-9379addb2585 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f4d29c290aa048f591948a8915e802fd'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:56.725477 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-628bfc87-743f-4520-a3f7-9379addb2585 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f4d29c290aa048f591948a8915e802fd'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:56.732854 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-628bfc87-743f-4520-a3f7-9379addb2585 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f4d29c290aa048f591948a8915e802fd'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:56.734901 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=83,keystone:DELETE=15,keystone:INSERT=2 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:56:56.740574 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 938/1847] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 00:56:56 2022] DELETE /identity/v3/users/098962dfac444426a5d60f7fca1f3efd => generated 0 bytes in 108 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:56:56.741422 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53850d3b-99dd-4f37-82b8-88ce642c4bf8 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:56.742767 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-53850d3b-99dd-4f37-82b8-88ce642c4bf8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:56.751739 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-628bfc87-743f-4520-a3f7-9379addb2585 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f4d29c290aa048f591948a8915e802fd'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:56.758831 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 910/1848] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 00:56:56 2022] DELETE /identity/v3/users/f4d29c290aa048f591948a8915e802fd => generated 0 bytes in 114 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:56:56.762683 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-628bfc87-743f-4520-a3f7-9379addb2585 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:56.764408 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-628bfc87-743f-4520-a3f7-9379addb2585 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:56.767729 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=85,keystone:INSERT=5,keystone:DELETE=10 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:56:56.860019 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-628bfc87-743f-4520-a3f7-9379addb2585 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:56.866947 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14096fe3-dc0e-42ad-a148-5f65505cb3fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:56.867968 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14096fe3-dc0e-42ad-a148-5f65505cb3fc admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:56.868520 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14096fe3-dc0e-42ad-a148-5f65505cb3fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:56.868803 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14096fe3-dc0e-42ad-a148-5f65505cb3fc admin admin] PATH_INFO: `/v3/projects` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:56.869467 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14096fe3-dc0e-42ad-a148-5f65505cb3fc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:56.872251 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14096fe3-dc0e-42ad-a148-5f65505cb3fc admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:56.880338 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53850d3b-99dd-4f37-82b8-88ce642c4bf8 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:56.894776 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2de1154c-609a-454e-af8e-f71c599e8e0b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:56.895609 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de1154c-609a-454e-af8e-f71c599e8e0b admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:56.895905 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de1154c-609a-454e-af8e-f71c599e8e0b admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:56.896240 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de1154c-609a-454e-af8e-f71c599e8e0b admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:56.896987 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2de1154c-609a-454e-af8e-f71c599e8e0b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:56.899358 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2de1154c-609a-454e-af8e-f71c599e8e0b admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:56.917713 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 911/1849] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 00:56:56 2022] POST /identity/v3/projects => generated 409 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:56:56.919730 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b97e468-96e9-400f-8596-836fe79571f7 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:56.920028 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b97e468-96e9-400f-8596-836fe79571f7 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:56.920310 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b97e468-96e9-400f-8596-836fe79571f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:56.932401 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 939/1850] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:56:56 2022] GET /identity/v3/roles => generated 1420 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:56:56.932401 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2de1154c-609a-454e-af8e-f71c599e8e0b admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:56.957482 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2de1154c-609a-454e-af8e-f71c599e8e0b admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:56.967628 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-1b97e468-96e9-400f-8596-836fe79571f7 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:56:56.968888 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-1b97e468-96e9-400f-8596-836fe79571f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:57.058022 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 912/1851] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:56:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:56:57.059090 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b97e468-96e9-400f-8596-836fe79571f7 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:57.060885 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-1b97e468-96e9-400f-8596-836fe79571f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:57.070480 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c10cf67b-2c99-48c2-928c-38931e2408da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:57.071239 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c10cf67b-2c99-48c2-928c-38931e2408da admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:57.071617 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c10cf67b-2c99-48c2-928c-38931e2408da admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:57.077798 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c10cf67b-2c99-48c2-928c-38931e2408da admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:57.077798 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c10cf67b-2c99-48c2-928c-38931e2408da admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:57.077798 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c10cf67b-2c99-48c2-928c-38931e2408da admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:57.088448 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b97e468-96e9-400f-8596-836fe79571f7 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:57.094398 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72281bc1-f2f1-49ce-b9ae-8087281c14c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:57.095166 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72281bc1-f2f1-49ce-b9ae-8087281c14c1 admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:57.095556 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72281bc1-f2f1-49ce-b9ae-8087281c14c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:57.095907 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72281bc1-f2f1-49ce-b9ae-8087281c14c1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:57.097133 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72281bc1-f2f1-49ce-b9ae-8087281c14c1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:57.098993 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72281bc1-f2f1-49ce-b9ae-8087281c14c1 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:57.111489 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 913/1852] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:56:57 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:56:57.112699 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 940/1853] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:56:56 2022] POST /identity/v3/users => generated 311 bytes in 181 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:56:57.113048 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72281bc1-f2f1-49ce-b9ae-8087281c14c1 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:57.114976 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-72281bc1-f2f1-49ce-b9ae-8087281c14c1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:57.119346 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c10cf67b-2c99-48c2-928c-38931e2408da admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:57.134505 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c10cf67b-2c99-48c2-928c-38931e2408da admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:57.140051 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-198e4c42-0758-4829-b230-dcc5257c01dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:57.140881 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-198e4c42-0758-4829-b230-dcc5257c01dc admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:57.141095 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-198e4c42-0758-4829-b230-dcc5257c01dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:57.141334 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-198e4c42-0758-4829-b230-dcc5257c01dc admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:57.142348 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-198e4c42-0758-4829-b230-dcc5257c01dc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:57.144072 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-198e4c42-0758-4829-b230-dcc5257c01dc admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:57.158361 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 941/1854] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:56:57 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:56:57.163416 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-198e4c42-0758-4829-b230-dcc5257c01dc admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:57.181391 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-198e4c42-0758-4829-b230-dcc5257c01dc admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:57.202550 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e922e64-fcbe-48e6-9132-30a044b02c9f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:57.203710 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e922e64-fcbe-48e6-9132-30a044b02c9f admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:57.203996 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e922e64-fcbe-48e6-9132-30a044b02c9f admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:57.204272 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e922e64-fcbe-48e6-9132-30a044b02c9f admin admin] PATH_INFO: `/v3/projects/02c483e0e68343e0be05cac138769c2c/users/3fb6d6206a8546b796008783aa1abcae/roles/588016933bab436d8b7c3ac76827356d` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:57.237703 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72281bc1-f2f1-49ce-b9ae-8087281c14c1 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:57.258423 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b916bb37-cbd8-495f-a827-ae4c0c92f9e3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:57.259047 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b916bb37-cbd8-495f-a827-ae4c0c92f9e3 service neutron] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:57.259331 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b916bb37-cbd8-495f-a827-ae4c0c92f9e3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:57.259625 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b916bb37-cbd8-495f-a827-ae4c0c92f9e3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:57.262706 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e922e64-fcbe-48e6-9132-30a044b02c9f admin admin] RBAC: Authorizing `identity:create_grant(project_id=02c483e0e68343e0be05cac138769c2c, user_id=3fb6d6206a8546b796008783aa1abcae, role_id=588016933bab436d8b7c3ac76827356d)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:57.266756 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e922e64-fcbe-48e6-9132-30a044b02c9f admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:57.273332 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b916bb37-cbd8-495f-a827-ae4c0c92f9e3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:57.274968 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b916bb37-cbd8-495f-a827-ae4c0c92f9e3 service neutron] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:57.281718 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=97,keystone:DELETE=2,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:56:57.295545 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 942/1855] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:56:57 2022] PUT /identity/v3/projects/02c483e0e68343e0be05cac138769c2c/users/3fb6d6206a8546b796008783aa1abcae/roles/588016933bab436d8b7c3ac76827356d => generated 0 bytes in 133 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:56:57.295900 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 914/1856] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:56:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 184 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:56:57.296688 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e922e64-fcbe-48e6-9132-30a044b02c9f admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:57.308858 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a14f13f-c404-4f9e-ab3e-88f6b153644b None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:57.309092 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a14f13f-c404-4f9e-ab3e-88f6b153644b None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:57.309471 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a14f13f-c404-4f9e-ab3e-88f6b153644b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:57.369096 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-3a14f13f-c404-4f9e-ab3e-88f6b153644b None None] MFA Rules not processed for user `3fb6d6206a8546b796008783aa1abcae`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:56:57.369096 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-3a14f13f-c404-4f9e-ab3e-88f6b153644b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:57.459554 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e922e64-fcbe-48e6-9132-30a044b02c9f admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:57.464575 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 915/1857] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:56:57 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:56:57.465689 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a14f13f-c404-4f9e-ab3e-88f6b153644b None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:57.468880 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-3a14f13f-c404-4f9e-ab3e-88f6b153644b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:57.485751 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a14f13f-c404-4f9e-ab3e-88f6b153644b None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:57.485918 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cff546f-6d31-4caf-a5f3-2ecd24340bf3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:57.487451 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cff546f-6d31-4caf-a5f3-2ecd24340bf3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:57.487795 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cff546f-6d31-4caf-a5f3-2ecd24340bf3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:57.488588 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cff546f-6d31-4caf-a5f3-2ecd24340bf3 admin admin] PATH_INFO: `/v3/projects/69bf73bbee1444febc675e813fd9af0c/users/4fae9d1c91c046378b08c390d58da7d0/roles/092036ee46ac427baf4d56468575f697` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:57.493170 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8526116e-ce61-484c-b2df-9704f74a3f4f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:57.494000 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8526116e-ce61-484c-b2df-9704f74a3f4f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:57.494647 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8526116e-ce61-484c-b2df-9704f74a3f4f admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:57.494647 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8526116e-ce61-484c-b2df-9704f74a3f4f admin admin] PATH_INFO: `/v3/projects/9201690b2e83499ea2a735e6c673b8af` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:57.496214 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8526116e-ce61-484c-b2df-9704f74a3f4f admin admin] RBAC: Authorizing `identity:delete_project(project_id=9201690b2e83499ea2a735e6c673b8af)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:57.498678 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8526116e-ce61-484c-b2df-9704f74a3f4f admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:57.575268 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cff546f-6d31-4caf-a5f3-2ecd24340bf3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=69bf73bbee1444febc675e813fd9af0c, user_id=4fae9d1c91c046378b08c390d58da7d0, role_id=092036ee46ac427baf4d56468575f697)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:57.578219 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cff546f-6d31-4caf-a5f3-2ecd24340bf3 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:57.581800 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:56:57.613510 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 943/1858] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:56:57 2022] PUT /identity/v3/projects/69bf73bbee1444febc675e813fd9af0c/users/4fae9d1c91c046378b08c390d58da7d0/roles/092036ee46ac427baf4d56468575f697 => generated 0 bytes in 317 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:56:57.614542 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cff546f-6d31-4caf-a5f3-2ecd24340bf3 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:57.623780 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-8526116e-ce61-484c-b2df-9704f74a3f4f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9201690b2e83499ea2a735e6c673b8af', 'request_id': 'req-8526116e-ce61-484c-b2df-9704f74a3f4f'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:57.631997 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-8526116e-ce61-484c-b2df-9704f74a3f4f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9201690b2e83499ea2a735e6c673b8af', 'request_id': 'req-8526116e-ce61-484c-b2df-9704f74a3f4f'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:57.639824 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-8526116e-ce61-484c-b2df-9704f74a3f4f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9201690b2e83499ea2a735e6c673b8af', 'request_id': 'req-8526116e-ce61-484c-b2df-9704f74a3f4f'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:57.668616 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-8526116e-ce61-484c-b2df-9704f74a3f4f admin admin] The token cache is being invalidate because project 9201690b2e83499ea2a735e6c673b8af was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81453) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 00:56:57.669203 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-8526116e-ce61-484c-b2df-9704f74a3f4f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:57.673486 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-8526116e-ce61-484c-b2df-9704f74a3f4f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:57.676965 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 916/1859] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 00:56:57 2022] DELETE /identity/v3/projects/9201690b2e83499ea2a735e6c673b8af => generated 0 bytes in 212 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:56:57.677809 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8526116e-ce61-484c-b2df-9704f74a3f4f admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:57.679225 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-8526116e-ce61-484c-b2df-9704f74a3f4f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:57.736488 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cff546f-6d31-4caf-a5f3-2ecd24340bf3 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:57.758102 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8526116e-ce61-484c-b2df-9704f74a3f4f admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:57.764051 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c81adb0-6908-4968-a3cc-0b90d2ce2471 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:57.766431 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c81adb0-6908-4968-a3cc-0b90d2ce2471 admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:57.766722 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c81adb0-6908-4968-a3cc-0b90d2ce2471 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:57.767003 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c81adb0-6908-4968-a3cc-0b90d2ce2471 admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:57.767675 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c81adb0-6908-4968-a3cc-0b90d2ce2471 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:57.771052 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c81adb0-6908-4968-a3cc-0b90d2ce2471 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:57.783200 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 917/1860] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:56:57 2022] GET /identity/v3/roles => generated 1420 bytes in 106 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:56:57.783200 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b64d2358-d34c-42a4-b55d-c583b8cfdc18 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:57.783200 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b64d2358-d34c-42a4-b55d-c583b8cfdc18 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:57.783200 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b64d2358-d34c-42a4-b55d-c583b8cfdc18 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:57.805934 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1d491aa-43a9-4cbd-915a-fd8157a025f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:57.806884 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d491aa-43a9-4cbd-915a-fd8157a025f7 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:57.807008 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d491aa-43a9-4cbd-915a-fd8157a025f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:57.807552 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d491aa-43a9-4cbd-915a-fd8157a025f7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:57.808147 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1d491aa-43a9-4cbd-915a-fd8157a025f7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:57.810318 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1d491aa-43a9-4cbd-915a-fd8157a025f7 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:57.812742 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-b64d2358-d34c-42a4-b55d-c583b8cfdc18 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:56:57.813960 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-b64d2358-d34c-42a4-b55d-c583b8cfdc18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:57.823068 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 918/1861] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:56:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:56:57.824764 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b64d2358-d34c-42a4-b55d-c583b8cfdc18 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:57.829790 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:56:57.836413 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b64d2358-d34c-42a4-b55d-c583b8cfdc18 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:57.845214 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daca6237-11b7-471d-8539-eaad18001f78 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:57.846050 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daca6237-11b7-471d-8539-eaad18001f78 admin admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:57.846936 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daca6237-11b7-471d-8539-eaad18001f78 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:57.847388 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daca6237-11b7-471d-8539-eaad18001f78 admin admin] PATH_INFO: `/v3/projects/69bf73bbee1444febc675e813fd9af0c/users/4fae9d1c91c046378b08c390d58da7d0/roles/092036ee46ac427baf4d56468575f697` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:57.851978 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daca6237-11b7-471d-8539-eaad18001f78 admin admin] RBAC: Authorizing `identity:create_grant(project_id=69bf73bbee1444febc675e813fd9af0c, user_id=4fae9d1c91c046378b08c390d58da7d0, role_id=092036ee46ac427baf4d56468575f697)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:57.855688 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daca6237-11b7-471d-8539-eaad18001f78 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:57.861655 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 944/1862] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 00:56:57 2022] POST /identity/v3/projects => generated 385 bytes in 246 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:56:57.864822 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9cf2e76-04b6-4077-bd18-36f273a38654 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:57.865147 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9cf2e76-04b6-4077-bd18-36f273a38654 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:57.865400 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9cf2e76-04b6-4077-bd18-36f273a38654 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:57.880080 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 919/1863] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:56:57 2022] PUT /identity/v3/projects/69bf73bbee1444febc675e813fd9af0c/users/4fae9d1c91c046378b08c390d58da7d0/roles/092036ee46ac427baf4d56468575f697 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:56:57.880931 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daca6237-11b7-471d-8539-eaad18001f78 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:57.892293 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-daca6237-11b7-471d-8539-eaad18001f78 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:57.897119 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-d9cf2e76-04b6-4077-bd18-36f273a38654 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:56:57.898394 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-d9cf2e76-04b6-4077-bd18-36f273a38654 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:57.996603 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 945/1864] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:56:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:56:57.997442 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daca6237-11b7-471d-8539-eaad18001f78 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:57.997819 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9cf2e76-04b6-4077-bd18-36f273a38654 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:57.999280 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-d9cf2e76-04b6-4077-bd18-36f273a38654 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:58.004531 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-921d3a1a-0008-45a9-bbc0-0fed599498ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:58.005426 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-921d3a1a-0008-45a9-bbc0-0fed599498ac admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:58.005783 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-921d3a1a-0008-45a9-bbc0-0fed599498ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:58.006120 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-921d3a1a-0008-45a9-bbc0-0fed599498ac admin admin] PATH_INFO: `/v3/projects/9619db18b00c4670a6bb50e4d3b982ef` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:58.007559 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-921d3a1a-0008-45a9-bbc0-0fed599498ac admin admin] RBAC: Authorizing `identity:delete_project(project_id=9619db18b00c4670a6bb50e4d3b982ef)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:58.010241 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-921d3a1a-0008-45a9-bbc0-0fed599498ac admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:58.013143 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9cf2e76-04b6-4077-bd18-36f273a38654 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:58.018584 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96a3c1b2-4244-4463-83ba-12c854048495 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:58.019189 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96a3c1b2-4244-4463-83ba-12c854048495 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:58.019456 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96a3c1b2-4244-4463-83ba-12c854048495 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:58.019765 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96a3c1b2-4244-4463-83ba-12c854048495 admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:58.020834 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96a3c1b2-4244-4463-83ba-12c854048495 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:58.022756 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96a3c1b2-4244-4463-83ba-12c854048495 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:58.051297 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 946/1865] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:56:57 2022] POST /identity/v3/users => generated 299 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:56:58.051297 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7449bf34-b481-4892-bc16-84983d2b507d None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:58.051297 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7449bf34-b481-4892-bc16-84983d2b507d None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:58.051297 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7449bf34-b481-4892-bc16-84983d2b507d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:58.072345 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:INSERT=5,keystone:SELECT=95 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:56:58.076527 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-921d3a1a-0008-45a9-bbc0-0fed599498ac admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9619db18b00c4670a6bb50e4d3b982ef', 'request_id': 'req-921d3a1a-0008-45a9-bbc0-0fed599498ac'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:58.089871 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-7449bf34-b481-4892-bc16-84983d2b507d None None] MFA Rules not processed for user `4fae9d1c91c046378b08c390d58da7d0`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:56:58.091113 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-7449bf34-b481-4892-bc16-84983d2b507d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:58.094026 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-921d3a1a-0008-45a9-bbc0-0fed599498ac admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9619db18b00c4670a6bb50e4d3b982ef', 'request_id': 'req-921d3a1a-0008-45a9-bbc0-0fed599498ac'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:58.101759 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-921d3a1a-0008-45a9-bbc0-0fed599498ac admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9619db18b00c4670a6bb50e4d3b982ef', 'request_id': 'req-921d3a1a-0008-45a9-bbc0-0fed599498ac'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:58.125125 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-921d3a1a-0008-45a9-bbc0-0fed599498ac admin admin] The token cache is being invalidate because project 9619db18b00c4670a6bb50e4d3b982ef was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81453) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 00:56:58.125660 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-921d3a1a-0008-45a9-bbc0-0fed599498ac admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:58.126927 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-921d3a1a-0008-45a9-bbc0-0fed599498ac admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:58.130211 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 920/1866] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 00:56:57 2022] DELETE /identity/v3/projects/9619db18b00c4670a6bb50e4d3b982ef => generated 0 bytes in 250 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:56:58.132019 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-921d3a1a-0008-45a9-bbc0-0fed599498ac admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:58.135558 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-921d3a1a-0008-45a9-bbc0-0fed599498ac admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:58.174127 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 947/1867] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:56:58 2022] POST /identity/v3/auth/tokens => generated 3353 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:56:58.175460 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7449bf34-b481-4892-bc16-84983d2b507d None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:58.179471 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-7449bf34-b481-4892-bc16-84983d2b507d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:58.220766 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-921d3a1a-0008-45a9-bbc0-0fed599498ac admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:58.226172 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4f0c5bc-070e-46bc-85df-3c2ce043eb79 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:58.226782 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f0c5bc-070e-46bc-85df-3c2ce043eb79 admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:58.227097 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f0c5bc-070e-46bc-85df-3c2ce043eb79 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:58.227340 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f0c5bc-070e-46bc-85df-3c2ce043eb79 admin admin] PATH_INFO: `/v3/domains` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:58.228025 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4f0c5bc-070e-46bc-85df-3c2ce043eb79 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:56:58.228329 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4f0c5bc-070e-46bc-85df-3c2ce043eb79 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:58.229921 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4f0c5bc-070e-46bc-85df-3c2ce043eb79 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:58.243786 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 921/1868] 10.222.0.33 () {58 vars in 1153 bytes} [Sun Oct 9 00:56:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 113 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:56:58.244617 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4f0c5bc-070e-46bc-85df-3c2ce043eb79 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:58.248408 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-f4f0c5bc-070e-46bc-85df-3c2ce043eb79 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:58.260411 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7449bf34-b481-4892-bc16-84983d2b507d None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:58.262843 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4f0c5bc-070e-46bc-85df-3c2ce043eb79 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:58.265487 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a8d776c-a867-4c02-b030-bedaee39b8af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:58.266095 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a8d776c-a867-4c02-b030-bedaee39b8af admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:58.266369 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a8d776c-a867-4c02-b030-bedaee39b8af admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:58.266741 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a8d776c-a867-4c02-b030-bedaee39b8af admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:58.267258 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a8d776c-a867-4c02-b030-bedaee39b8af admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:58.267520 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-159baa2b-054a-44fd-9681-a5416445bdb3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:58.268760 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-159baa2b-054a-44fd-9681-a5416445bdb3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:58.269048 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-159baa2b-054a-44fd-9681-a5416445bdb3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:58.269323 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-159baa2b-054a-44fd-9681-a5416445bdb3 admin admin] PATH_INFO: `/v3/users/243bcff947fc4fd59e32dbc9cb272868` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:58.269432 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a8d776c-a867-4c02-b030-bedaee39b8af admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:58.270577 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-159baa2b-054a-44fd-9681-a5416445bdb3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=243bcff947fc4fd59e32dbc9cb272868)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:58.272519 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-159baa2b-054a-44fd-9681-a5416445bdb3 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:58.279053 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:56:58.280405 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 948/1869] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:56:58 2022] GET /identity/v3/roles => generated 1420 bytes in 106 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:56:58.281689 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a8d776c-a867-4c02-b030-bedaee39b8af admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:58.283642 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-9a8d776c-a867-4c02-b030-bedaee39b8af admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:58.355253 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-159baa2b-054a-44fd-9681-a5416445bdb3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '243bcff947fc4fd59e32dbc9cb272868'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:58.360415 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a8d776c-a867-4c02-b030-bedaee39b8af admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:58.363995 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-159baa2b-054a-44fd-9681-a5416445bdb3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '243bcff947fc4fd59e32dbc9cb272868'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:58.364702 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-159baa2b-054a-44fd-9681-a5416445bdb3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '243bcff947fc4fd59e32dbc9cb272868'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:58.369154 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82425cec-7bbb-4935-af84-61341060e1ce service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3f48335668a4e8f934ac11ff66cd308', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:58.369956 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82425cec-7bbb-4935-af84-61341060e1ce service cinder] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:58.370361 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82425cec-7bbb-4935-af84-61341060e1ce service cinder] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:58.370729 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82425cec-7bbb-4935-af84-61341060e1ce service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:58.372449 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-159baa2b-054a-44fd-9681-a5416445bdb3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '243bcff947fc4fd59e32dbc9cb272868'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:58.373051 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-82425cec-7bbb-4935-af84-61341060e1ce service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:58.385412 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-159baa2b-054a-44fd-9681-a5416445bdb3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '243bcff947fc4fd59e32dbc9cb272868'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:58.387884 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82425cec-7bbb-4935-af84-61341060e1ce service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:58.389469 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82425cec-7bbb-4935-af84-61341060e1ce service cinder] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:58.391874 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 922/1870] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 00:56:58 2022] DELETE /identity/v3/users/243bcff947fc4fd59e32dbc9cb272868 => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:56:58.392690 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-159baa2b-054a-44fd-9681-a5416445bdb3 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:58.394795 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-159baa2b-054a-44fd-9681-a5416445bdb3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:58.403582 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 949/1871] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:56:58 2022] GET /identity/v3/auth/tokens => generated 3353 bytes in 123 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:56:58.404499 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82425cec-7bbb-4935-af84-61341060e1ce service cinder] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:58.406405 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-82425cec-7bbb-4935-af84-61341060e1ce service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:58.421982 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82425cec-7bbb-4935-af84-61341060e1ce service cinder] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:58.474196 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:56:58.512085 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-159baa2b-054a-44fd-9681-a5416445bdb3 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:58.512778 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96073835-eab7-4ea6-92f4-a681ba018479 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:58.513660 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96073835-eab7-4ea6-92f4-a681ba018479 admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:58.514586 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96073835-eab7-4ea6-92f4-a681ba018479 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:58.515479 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96073835-eab7-4ea6-92f4-a681ba018479 admin admin] PATH_INFO: `/v3/projects/ad3618890d6848a781503ee33472cf7a/users/05a49698531a4efab9bc686b0415eb14/roles/092036ee46ac427baf4d56468575f697` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:58.517983 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1876f7ca-32fd-4e1b-9f72-5f923c117d3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:58.519328 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1876f7ca-32fd-4e1b-9f72-5f923c117d3d admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:58.519747 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1876f7ca-32fd-4e1b-9f72-5f923c117d3d admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:58.520545 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1876f7ca-32fd-4e1b-9f72-5f923c117d3d admin admin] PATH_INFO: `/v3/projects` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:58.522461 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1876f7ca-32fd-4e1b-9f72-5f923c117d3d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:58.524944 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1876f7ca-32fd-4e1b-9f72-5f923c117d3d admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:58.540701 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96073835-eab7-4ea6-92f4-a681ba018479 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ad3618890d6848a781503ee33472cf7a, user_id=05a49698531a4efab9bc686b0415eb14, role_id=092036ee46ac427baf4d56468575f697)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:58.544723 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96073835-eab7-4ea6-92f4-a681ba018479 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:58.559937 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 923/1872] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 00:56:58 2022] POST /identity/v3/projects => generated 385 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:56:58.564132 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1876f7ca-32fd-4e1b-9f72-5f923c117d3d admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:58.564706 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 950/1873] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:56:58 2022] PUT /identity/v3/projects/ad3618890d6848a781503ee33472cf7a/users/05a49698531a4efab9bc686b0415eb14/roles/092036ee46ac427baf4d56468575f697 => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:56:58.569060 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96073835-eab7-4ea6-92f4-a681ba018479 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:58.656352 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96073835-eab7-4ea6-92f4-a681ba018479 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:58.657302 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1876f7ca-32fd-4e1b-9f72-5f923c117d3d admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:58.662566 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65e01c84-b23b-427d-b928-0cc203f52644 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:58.662753 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94eb8984-1d62-4d38-81d1-1c7592263047 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:58.663178 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65e01c84-b23b-427d-b928-0cc203f52644 admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:58.663461 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65e01c84-b23b-427d-b928-0cc203f52644 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:58.663769 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65e01c84-b23b-427d-b928-0cc203f52644 admin admin] PATH_INFO: `/v3/users` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:58.663913 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94eb8984-1d62-4d38-81d1-1c7592263047 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:58.664309 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94eb8984-1d62-4d38-81d1-1c7592263047 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:58.664428 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65e01c84-b23b-427d-b928-0cc203f52644 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:58.664747 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94eb8984-1d62-4d38-81d1-1c7592263047 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:58.665683 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94eb8984-1d62-4d38-81d1-1c7592263047 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:58.666438 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65e01c84-b23b-427d-b928-0cc203f52644 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:58.667525 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94eb8984-1d62-4d38-81d1-1c7592263047 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:58.678727 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 951/1874] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:56:58 2022] GET /identity/v3/roles => generated 1420 bytes in 110 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 09 00:56:58.682881 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94eb8984-1d62-4d38-81d1-1c7592263047 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:58.684750 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-94eb8984-1d62-4d38-81d1-1c7592263047 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:58.696964 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=85,keystone:DELETE=10,keystone:INSERT=5 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:56:58.704820 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 924/1875] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:56:58 2022] POST /identity/v3/users => generated 293 bytes in 141 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 09 00:56:58.706460 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65e01c84-b23b-427d-b928-0cc203f52644 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:58.718496 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65e01c84-b23b-427d-b928-0cc203f52644 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:58.725298 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26c6c542-f39d-43a7-a865-4d26bf13255c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:58.726641 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26c6c542-f39d-43a7-a865-4d26bf13255c admin admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:58.727020 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26c6c542-f39d-43a7-a865-4d26bf13255c admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:58.727390 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26c6c542-f39d-43a7-a865-4d26bf13255c admin admin] PATH_INFO: `/v3/projects/ad3618890d6848a781503ee33472cf7a/users/05a49698531a4efab9bc686b0415eb14/roles/092036ee46ac427baf4d56468575f697` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:58.730365 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26c6c542-f39d-43a7-a865-4d26bf13255c admin admin] RBAC: Authorizing `identity:create_grant(project_id=ad3618890d6848a781503ee33472cf7a, user_id=05a49698531a4efab9bc686b0415eb14, role_id=092036ee46ac427baf4d56468575f697)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:58.733450 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26c6c542-f39d-43a7-a865-4d26bf13255c admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:58.754088 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 925/1876] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:56:58 2022] PUT /identity/v3/projects/ad3618890d6848a781503ee33472cf7a/users/05a49698531a4efab9bc686b0415eb14/roles/092036ee46ac427baf4d56468575f697 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:56:58.755006 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26c6c542-f39d-43a7-a865-4d26bf13255c admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:58.783331 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94eb8984-1d62-4d38-81d1-1c7592263047 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:58.856550 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26c6c542-f39d-43a7-a865-4d26bf13255c admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:58.861403 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd55dd2a-05b6-4f8a-8d89-dec827fefdd7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:58.862022 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd55dd2a-05b6-4f8a-8d89-dec827fefdd7 admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:58.862302 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd55dd2a-05b6-4f8a-8d89-dec827fefdd7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:58.862570 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd55dd2a-05b6-4f8a-8d89-dec827fefdd7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:58.863319 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd55dd2a-05b6-4f8a-8d89-dec827fefdd7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:58.864945 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd55dd2a-05b6-4f8a-8d89-dec827fefdd7 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:58.872081 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-055012be-8b84-4778-8f25-25f3365a9035 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3f48335668a4e8f934ac11ff66cd308', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:58.872875 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-055012be-8b84-4778-8f25-25f3365a9035 service cinder] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:58.873254 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-055012be-8b84-4778-8f25-25f3365a9035 service cinder] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:58.873628 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-055012be-8b84-4778-8f25-25f3365a9035 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:58.875315 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 926/1877] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:56:58 2022] GET /identity/v3/roles => generated 1420 bytes in 120 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:56:58.876458 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4a91c97-12b2-489d-b192-746092c4516f None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:58.876759 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4a91c97-12b2-489d-b192-746092c4516f None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:58.877019 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4a91c97-12b2-489d-b192-746092c4516f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:58.884432 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-055012be-8b84-4778-8f25-25f3365a9035 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:58.915472 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-f4a91c97-12b2-489d-b192-746092c4516f None None] MFA Rules not processed for user `05a49698531a4efab9bc686b0415eb14`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:56:58.916912 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-f4a91c97-12b2-489d-b192-746092c4516f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:58.919875 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:56:58.972978 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-055012be-8b84-4778-8f25-25f3365a9035 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:58.975176 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-055012be-8b84-4778-8f25-25f3365a9035 service cinder] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:58.988489 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 952/1878] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:56:58 2022] GET /identity/v3/auth/tokens => generated 3284 bytes in 306 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:56:58.989404 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-055012be-8b84-4778-8f25-25f3365a9035 service cinder] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:59.001262 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-055012be-8b84-4778-8f25-25f3365a9035 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:59.001590 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 927/1879] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:56:58 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:56:59.002554 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4a91c97-12b2-489d-b192-746092c4516f None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:59.019712 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-055012be-8b84-4778-8f25-25f3365a9035 service cinder] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:59.024654 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4a91c97-12b2-489d-b192-746092c4516f None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:59.025587 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4436ca9-e4d3-4c45-a7cd-20bfcbdd9b8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:59.026234 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4436ca9-e4d3-4c45-a7cd-20bfcbdd9b8e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:59.026548 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4436ca9-e4d3-4c45-a7cd-20bfcbdd9b8e admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:59.026915 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4436ca9-e4d3-4c45-a7cd-20bfcbdd9b8e admin admin] PATH_INFO: `/v3/projects/63fe6805f72c48ef817bef1b2cf2a37b` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:59.028147 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4436ca9-e4d3-4c45-a7cd-20bfcbdd9b8e admin admin] RBAC: Authorizing `identity:delete_project(project_id=63fe6805f72c48ef817bef1b2cf2a37b)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:59.030164 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4436ca9-e4d3-4c45-a7cd-20bfcbdd9b8e admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:59.032635 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68f7da3c-883a-4cb5-924f-d820a33b5c0b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:59.033360 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68f7da3c-883a-4cb5-924f-d820a33b5c0b admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:59.033768 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68f7da3c-883a-4cb5-924f-d820a33b5c0b admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:59.034160 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68f7da3c-883a-4cb5-924f-d820a33b5c0b admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:59.034897 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68f7da3c-883a-4cb5-924f-d820a33b5c0b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:59.036854 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68f7da3c-883a-4cb5-924f-d820a33b5c0b admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:59.052553 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 928/1880] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:56:59 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:56:59.056563 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68f7da3c-883a-4cb5-924f-d820a33b5c0b admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:59.068442 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68f7da3c-883a-4cb5-924f-d820a33b5c0b admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:59.087410 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8856d032-26e6-4699-991d-63434159258b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:59.088694 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8856d032-26e6-4699-991d-63434159258b admin admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:59.088802 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8856d032-26e6-4699-991d-63434159258b admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:59.089162 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8856d032-26e6-4699-991d-63434159258b admin admin] PATH_INFO: `/v3/projects/bef1dcb3d43f44de8a3e058b4dee56ab/users/c8d21a27678b46d5aef210817abfd991/roles/588016933bab436d8b7c3ac76827356d` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:59.126294 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-e4436ca9-e4d3-4c45-a7cd-20bfcbdd9b8e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '63fe6805f72c48ef817bef1b2cf2a37b', 'request_id': 'req-e4436ca9-e4d3-4c45-a7cd-20bfcbdd9b8e'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:59.139268 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8856d032-26e6-4699-991d-63434159258b admin admin] RBAC: Authorizing `identity:create_grant(project_id=bef1dcb3d43f44de8a3e058b4dee56ab, user_id=c8d21a27678b46d5aef210817abfd991, role_id=588016933bab436d8b7c3ac76827356d)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:59.148783 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8856d032-26e6-4699-991d-63434159258b admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:59.149078 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-e4436ca9-e4d3-4c45-a7cd-20bfcbdd9b8e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '63fe6805f72c48ef817bef1b2cf2a37b', 'request_id': 'req-e4436ca9-e4d3-4c45-a7cd-20bfcbdd9b8e'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:59.172583 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-e4436ca9-e4d3-4c45-a7cd-20bfcbdd9b8e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '63fe6805f72c48ef817bef1b2cf2a37b', 'request_id': 'req-e4436ca9-e4d3-4c45-a7cd-20bfcbdd9b8e'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:59.191549 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 929/1881] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:56:59 2022] PUT /identity/v3/projects/bef1dcb3d43f44de8a3e058b4dee56ab/users/c8d21a27678b46d5aef210817abfd991/roles/588016933bab436d8b7c3ac76827356d => generated 0 bytes in 133 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:56:59.191549 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8856d032-26e6-4699-991d-63434159258b admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:59.200797 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-8856d032-26e6-4699-991d-63434159258b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:59.214723 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-e4436ca9-e4d3-4c45-a7cd-20bfcbdd9b8e admin admin] The token cache is being invalidate because project 63fe6805f72c48ef817bef1b2cf2a37b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 00:56:59.216349 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-e4436ca9-e4d3-4c45-a7cd-20bfcbdd9b8e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:59.218520 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-e4436ca9-e4d3-4c45-a7cd-20bfcbdd9b8e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:56:59.221311 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 953/1882] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 00:56:58 2022] DELETE /identity/v3/projects/63fe6805f72c48ef817bef1b2cf2a37b => generated 0 bytes in 232 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:56:59.224381 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb632f4-c050-4860-b64e-a50210d2e589 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:59.224861 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb632f4-c050-4860-b64e-a50210d2e589 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:59.225286 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb632f4-c050-4860-b64e-a50210d2e589 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:59.304876 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-2bb632f4-c050-4860-b64e-a50210d2e589 None None] MFA Rules not processed for user `c8d21a27678b46d5aef210817abfd991`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:56:59.306618 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-2bb632f4-c050-4860-b64e-a50210d2e589 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:59.307739 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:56:59.344895 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8856d032-26e6-4699-991d-63434159258b admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:59.362582 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e6de387-b13c-4459-b1f5-c7baf95344b5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3f48335668a4e8f934ac11ff66cd308', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:59.363507 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e6de387-b13c-4459-b1f5-c7baf95344b5 service cinder] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:59.364018 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e6de387-b13c-4459-b1f5-c7baf95344b5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:59.364656 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e6de387-b13c-4459-b1f5-c7baf95344b5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:59.374833 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-6e6de387-b13c-4459-b1f5-c7baf95344b5 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:59.472768 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 954/1883] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:56:59 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 249 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:56:59.474311 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fcf0cae-ac3f-48cc-8a67-ec538a6d52e2 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:59.474758 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fcf0cae-ac3f-48cc-8a67-ec538a6d52e2 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:59.475204 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fcf0cae-ac3f-48cc-8a67-ec538a6d52e2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:59.518012 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-2fcf0cae-ac3f-48cc-8a67-ec538a6d52e2 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:56:59.518313 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-2fcf0cae-ac3f-48cc-8a67-ec538a6d52e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:59.577962 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e6de387-b13c-4459-b1f5-c7baf95344b5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:59.586996 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e6de387-b13c-4459-b1f5-c7baf95344b5 service cinder] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:59.593712 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:56:59.627304 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 930/1884] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:56:59 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 436 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:56:59.629890 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e6de387-b13c-4459-b1f5-c7baf95344b5 service cinder] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:59.646234 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-6e6de387-b13c-4459-b1f5-c7baf95344b5 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:59.664987 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 955/1885] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:56:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:56:59.677651 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be7db919-e942-482e-806c-e0c8504b19c1 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:59.677938 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be7db919-e942-482e-806c-e0c8504b19c1 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:59.679626 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be7db919-e942-482e-806c-e0c8504b19c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:59.681385 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e6de387-b13c-4459-b1f5-c7baf95344b5 service cinder] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:59.686932 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0393dff6-55d8-45a2-b3d4-7fd207142a3f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:59.687697 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0393dff6-55d8-45a2-b3d4-7fd207142a3f admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:59.687979 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0393dff6-55d8-45a2-b3d4-7fd207142a3f admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:59.688250 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0393dff6-55d8-45a2-b3d4-7fd207142a3f admin admin] PATH_INFO: `/v3/projects` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:59.688971 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0393dff6-55d8-45a2-b3d4-7fd207142a3f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:59.691511 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0393dff6-55d8-45a2-b3d4-7fd207142a3f admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:59.744872 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 931/1886] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 00:56:59 2022] POST /identity/v3/projects => generated 385 bytes in 117 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:56:59.746586 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-be7db919-e942-482e-806c-e0c8504b19c1 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:56:59.748199 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-be7db919-e942-482e-806c-e0c8504b19c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:59.748976 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0393dff6-55d8-45a2-b3d4-7fd207142a3f admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:59.750922 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-0393dff6-55d8-45a2-b3d4-7fd207142a3f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:56:59.864284 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0393dff6-55d8-45a2-b3d4-7fd207142a3f admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:59.880466 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 956/1887] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:56:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 201 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:56:59.880754 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04b00770-7845-46e7-b68f-fa2c2305b922 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:59.880754 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04b00770-7845-46e7-b68f-fa2c2305b922 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:59.880754 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04b00770-7845-46e7-b68f-fa2c2305b922 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:59.881026 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04b00770-7845-46e7-b68f-fa2c2305b922 admin admin] PATH_INFO: `/v3/users/3fb6d6206a8546b796008783aa1abcae` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:59.882224 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be7db919-e942-482e-806c-e0c8504b19c1 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:59.883189 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04b00770-7845-46e7-b68f-fa2c2305b922 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3fb6d6206a8546b796008783aa1abcae)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:59.888875 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04b00770-7845-46e7-b68f-fa2c2305b922 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:59.906108 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be7db919-e942-482e-806c-e0c8504b19c1 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:59.920693 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24e7ae36-116d-48cf-a6ce-fff23c97d076 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:59.921544 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e7ae36-116d-48cf-a6ce-fff23c97d076 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:59.921835 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e7ae36-116d-48cf-a6ce-fff23c97d076 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:59.922110 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e7ae36-116d-48cf-a6ce-fff23c97d076 admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:59.923863 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24e7ae36-116d-48cf-a6ce-fff23c97d076 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:59.930719 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24e7ae36-116d-48cf-a6ce-fff23c97d076 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:56:59.962448 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 957/1888] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:56:59 2022] POST /identity/v3/users => generated 299 bytes in 85 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:56:59.966313 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24e7ae36-116d-48cf-a6ce-fff23c97d076 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:56:59.977988 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24e7ae36-116d-48cf-a6ce-fff23c97d076 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:56:59.984624 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0cc1233-a373-483e-87b0-99796bea21bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:56:59.985474 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0cc1233-a373-483e-87b0-99796bea21bc admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:56:59.985474 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0cc1233-a373-483e-87b0-99796bea21bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:56:59.985756 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0cc1233-a373-483e-87b0-99796bea21bc admin admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:56:59.986394 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0cc1233-a373-483e-87b0-99796bea21bc admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:56:59.986696 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0cc1233-a373-483e-87b0-99796bea21bc admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:56:59.989184 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0cc1233-a373-483e-87b0-99796bea21bc admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:00.007319 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 958/1889] 10.222.0.33 () {58 vars in 1153 bytes} [Sun Oct 9 00:56:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:57:00.008254 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0cc1233-a373-483e-87b0-99796bea21bc admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:00.023543 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-04b00770-7845-46e7-b68f-fa2c2305b922 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3fb6d6206a8546b796008783aa1abcae'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:00.034916 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-04b00770-7845-46e7-b68f-fa2c2305b922 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3fb6d6206a8546b796008783aa1abcae'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:00.036038 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0cc1233-a373-483e-87b0-99796bea21bc admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:00.036155 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-04b00770-7845-46e7-b68f-fa2c2305b922 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3fb6d6206a8546b796008783aa1abcae'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:00.046484 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b1378bd-907a-4c2b-be3b-1bad5221bf17 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:00.047500 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-04b00770-7845-46e7-b68f-fa2c2305b922 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3fb6d6206a8546b796008783aa1abcae'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:00.048724 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=90,keystone:INSERT=2,keystone:DELETE=8 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:00.048958 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b1378bd-907a-4c2b-be3b-1bad5221bf17 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:00.049242 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b1378bd-907a-4c2b-be3b-1bad5221bf17 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:00.050093 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b1378bd-907a-4c2b-be3b-1bad5221bf17 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:00.050747 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b1378bd-907a-4c2b-be3b-1bad5221bf17 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:00.052408 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b1378bd-907a-4c2b-be3b-1bad5221bf17 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:00.063715 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 959/1890] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:57:00 2022] GET /identity/v3/roles => generated 1420 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:57:00.064525 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b1378bd-907a-4c2b-be3b-1bad5221bf17 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:00.068247 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-6b1378bd-907a-4c2b-be3b-1bad5221bf17 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:00.069119 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-04b00770-7845-46e7-b68f-fa2c2305b922 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3fb6d6206a8546b796008783aa1abcae'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:00.082500 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b1378bd-907a-4c2b-be3b-1bad5221bf17 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:00.085030 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 932/1891] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 00:56:59 2022] DELETE /identity/v3/users/3fb6d6206a8546b796008783aa1abcae => generated 0 bytes in 340 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:00.086613 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04b00770-7845-46e7-b68f-fa2c2305b922 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:00.088264 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-04b00770-7845-46e7-b68f-fa2c2305b922 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:00.157550 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04b00770-7845-46e7-b68f-fa2c2305b922 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:00.162555 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43902406-d3be-4741-94c9-a33aee623c2c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:00.163220 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43902406-d3be-4741-94c9-a33aee623c2c admin admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:00.163498 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43902406-d3be-4741-94c9-a33aee623c2c admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:00.163902 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-232a1d57-3148-4b69-94ad-6e37959cf5e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:00.164079 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43902406-d3be-4741-94c9-a33aee623c2c admin admin] PATH_INFO: `/v3/projects/d4df28446e714a29b69ebc706ea6be93/users/647c3bb7f2fe44d39b6bdfed4a75a61a/roles/092036ee46ac427baf4d56468575f697` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:00.164846 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-232a1d57-3148-4b69-94ad-6e37959cf5e7 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:00.165266 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-232a1d57-3148-4b69-94ad-6e37959cf5e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:00.165688 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-232a1d57-3148-4b69-94ad-6e37959cf5e7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:00.166593 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-232a1d57-3148-4b69-94ad-6e37959cf5e7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:00.168820 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-232a1d57-3148-4b69-94ad-6e37959cf5e7 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:00.179848 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43902406-d3be-4741-94c9-a33aee623c2c admin admin] RBAC: Authorizing `identity:create_grant(project_id=d4df28446e714a29b69ebc706ea6be93, user_id=647c3bb7f2fe44d39b6bdfed4a75a61a, role_id=092036ee46ac427baf4d56468575f697)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:00.182502 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43902406-d3be-4741-94c9-a33aee623c2c admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:00.196780 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 933/1892] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:57:00 2022] PUT /identity/v3/projects/d4df28446e714a29b69ebc706ea6be93/users/647c3bb7f2fe44d39b6bdfed4a75a61a/roles/092036ee46ac427baf4d56468575f697 => generated 0 bytes in 111 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:00.198882 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e90640-11da-41d8-b8fb-ef5752dd9b32 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:00.198882 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e90640-11da-41d8-b8fb-ef5752dd9b32 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:00.198882 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e90640-11da-41d8-b8fb-ef5752dd9b32 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:00.203826 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 960/1893] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 00:57:00 2022] POST /identity/v3/projects => generated 381 bytes in 139 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:57:00.204612 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-232a1d57-3148-4b69-94ad-6e37959cf5e7 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:00.221780 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-71e90640-11da-41d8-b8fb-ef5752dd9b32 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:00.223041 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-71e90640-11da-41d8-b8fb-ef5752dd9b32 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:00.226480 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:00.278531 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-232a1d57-3148-4b69-94ad-6e37959cf5e7 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:00.283568 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-694cf4ce-1c9c-42f4-93d3-3111ef797f01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:00.284799 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-694cf4ce-1c9c-42f4-93d3-3111ef797f01 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:00.285083 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-694cf4ce-1c9c-42f4-93d3-3111ef797f01 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:00.285356 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-694cf4ce-1c9c-42f4-93d3-3111ef797f01 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:00.285999 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-694cf4ce-1c9c-42f4-93d3-3111ef797f01 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:00.287712 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-694cf4ce-1c9c-42f4-93d3-3111ef797f01 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:00.292255 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 934/1894] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:57:00.292766 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71e90640-11da-41d8-b8fb-ef5752dd9b32 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:00.300272 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 961/1895] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:57:00 2022] GET /identity/v3/roles => generated 1420 bytes in 96 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:57:00.303350 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71e90640-11da-41d8-b8fb-ef5752dd9b32 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:00.304179 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-694cf4ce-1c9c-42f4-93d3-3111ef797f01 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:00.309247 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0f2f861-f0b3-40d6-9c48-f259fa5426a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:00.309247 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f2f861-f0b3-40d6-9c48-f259fa5426a5 admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:00.309247 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f2f861-f0b3-40d6-9c48-f259fa5426a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:00.309577 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f2f861-f0b3-40d6-9c48-f259fa5426a5 admin admin] PATH_INFO: `/v3/users` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:00.310031 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0f2f861-f0b3-40d6-9c48-f259fa5426a5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:00.312331 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0f2f861-f0b3-40d6-9c48-f259fa5426a5 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:00.315019 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-694cf4ce-1c9c-42f4-93d3-3111ef797f01 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:00.333721 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9868358-ce28-4039-bc8c-db869988dc11 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:00.334660 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9868358-ce28-4039-bc8c-db869988dc11 admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:00.335139 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9868358-ce28-4039-bc8c-db869988dc11 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:00.335559 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9868358-ce28-4039-bc8c-db869988dc11 admin admin] PATH_INFO: `/v3/projects/d4df28446e714a29b69ebc706ea6be93/users/647c3bb7f2fe44d39b6bdfed4a75a61a/roles/092036ee46ac427baf4d56468575f697` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:00.339957 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9868358-ce28-4039-bc8c-db869988dc11 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d4df28446e714a29b69ebc706ea6be93, user_id=647c3bb7f2fe44d39b6bdfed4a75a61a, role_id=092036ee46ac427baf4d56468575f697)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:00.340618 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 935/1896] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:57:00 2022] POST /identity/v3/users => generated 291 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:57:00.342142 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0f2f861-f0b3-40d6-9c48-f259fa5426a5 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:00.344590 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9868358-ce28-4039-bc8c-db869988dc11 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:00.344868 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-b0f2f861-f0b3-40d6-9c48-f259fa5426a5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:00.364429 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 962/1897] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:57:00 2022] PUT /identity/v3/projects/d4df28446e714a29b69ebc706ea6be93/users/647c3bb7f2fe44d39b6bdfed4a75a61a/roles/092036ee46ac427baf4d56468575f697 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 09 00:57:00.365641 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9868358-ce28-4039-bc8c-db869988dc11 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:00.413624 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=94,keystone:DELETE=2,keystone:INSERT=4 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:00.419884 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0f2f861-f0b3-40d6-9c48-f259fa5426a5 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:00.451656 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9868358-ce28-4039-bc8c-db869988dc11 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:00.457697 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5585e795-2c39-4170-bac8-232c8b42f4ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:00.458926 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5585e795-2c39-4170-bac8-232c8b42f4ce admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:00.459208 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5585e795-2c39-4170-bac8-232c8b42f4ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:00.459541 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5585e795-2c39-4170-bac8-232c8b42f4ce admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:00.460191 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5585e795-2c39-4170-bac8-232c8b42f4ce admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:00.461870 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5585e795-2c39-4170-bac8-232c8b42f4ce admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:00.475099 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 963/1898] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:57:00 2022] GET /identity/v3/roles => generated 1420 bytes in 110 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:57:00.476485 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf68f100-b0d0-4ded-aff7-5a7dd5bfbf3d None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:00.476769 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf68f100-b0d0-4ded-aff7-5a7dd5bfbf3d None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:00.477041 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf68f100-b0d0-4ded-aff7-5a7dd5bfbf3d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:00.482551 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2cd7cd1-ba41-48ae-8c54-50794ad22e29 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:00.483193 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2cd7cd1-ba41-48ae-8c54-50794ad22e29 service neutron] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:00.483475 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2cd7cd1-ba41-48ae-8c54-50794ad22e29 service neutron] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:00.483788 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2cd7cd1-ba41-48ae-8c54-50794ad22e29 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:00.494012 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2cd7cd1-ba41-48ae-8c54-50794ad22e29 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:00.495471 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2cd7cd1-ba41-48ae-8c54-50794ad22e29 service neutron] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:00.505449 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 936/1899] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:57:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:57:00.506269 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2cd7cd1-ba41-48ae-8c54-50794ad22e29 service neutron] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:00.509412 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-bf68f100-b0d0-4ded-aff7-5a7dd5bfbf3d None None] MFA Rules not processed for user `647c3bb7f2fe44d39b6bdfed4a75a61a`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:00.510881 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-bf68f100-b0d0-4ded-aff7-5a7dd5bfbf3d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:00.516119 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2cd7cd1-ba41-48ae-8c54-50794ad22e29 service neutron] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:00.521563 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ee3d208-ea25-441b-9289-6fed9f2d7754 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:00.522204 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ee3d208-ea25-441b-9289-6fed9f2d7754 admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:00.522484 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ee3d208-ea25-441b-9289-6fed9f2d7754 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:00.522755 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ee3d208-ea25-441b-9289-6fed9f2d7754 admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:00.523508 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ee3d208-ea25-441b-9289-6fed9f2d7754 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:00.525192 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ee3d208-ea25-441b-9289-6fed9f2d7754 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:00.538577 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 937/1900] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:57:00 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:57:00.543161 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ee3d208-ea25-441b-9289-6fed9f2d7754 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:00.558783 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ee3d208-ea25-441b-9289-6fed9f2d7754 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:00.564015 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e207a4da-17d5-446d-8c1f-eb3653c9e6a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:00.564748 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e207a4da-17d5-446d-8c1f-eb3653c9e6a9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:00.565160 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e207a4da-17d5-446d-8c1f-eb3653c9e6a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:00.565557 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e207a4da-17d5-446d-8c1f-eb3653c9e6a9 admin admin] PATH_INFO: `/v3/projects/6824f2b206d44e33b37c484dcea0e96f/users/63c5754454dc43faa321fb13d863ada9/roles/588016933bab436d8b7c3ac76827356d` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:00.586047 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e207a4da-17d5-446d-8c1f-eb3653c9e6a9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6824f2b206d44e33b37c484dcea0e96f, user_id=63c5754454dc43faa321fb13d863ada9, role_id=588016933bab436d8b7c3ac76827356d)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:00.590768 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e207a4da-17d5-446d-8c1f-eb3653c9e6a9 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:00.612302 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 964/1901] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:00 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:57:00.628821 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 938/1902] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:57:00 2022] PUT /identity/v3/projects/6824f2b206d44e33b37c484dcea0e96f/users/63c5754454dc43faa321fb13d863ada9/roles/588016933bab436d8b7c3ac76827356d => generated 0 bytes in 76 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:00.629146 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf68f100-b0d0-4ded-aff7-5a7dd5bfbf3d None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:00.629146 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-bf68f100-b0d0-4ded-aff7-5a7dd5bfbf3d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:00.630108 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c90b8a77-45f7-4a7d-91d8-92fb16f2f6e9 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:00.630453 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c90b8a77-45f7-4a7d-91d8-92fb16f2f6e9 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:00.630727 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c90b8a77-45f7-4a7d-91d8-92fb16f2f6e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:00.666961 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:00.699629 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-c90b8a77-45f7-4a7d-91d8-92fb16f2f6e9 None None] MFA Rules not processed for user `63c5754454dc43faa321fb13d863ada9`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:00.699629 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-c90b8a77-45f7-4a7d-91d8-92fb16f2f6e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:00.736574 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf68f100-b0d0-4ded-aff7-5a7dd5bfbf3d None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:00.840712 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af6fa5cb-e1a2-44a6-9e9b-880281fdae7d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3f48335668a4e8f934ac11ff66cd308', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:00.843110 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af6fa5cb-e1a2-44a6-9e9b-880281fdae7d service cinder] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:00.844119 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af6fa5cb-e1a2-44a6-9e9b-880281fdae7d service cinder] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:00.845764 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af6fa5cb-e1a2-44a6-9e9b-880281fdae7d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:00.849499 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 939/1903] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:00 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 221 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 09 00:57:00.858432 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c90b8a77-45f7-4a7d-91d8-92fb16f2f6e9 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:00.955274 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c90b8a77-45f7-4a7d-91d8-92fb16f2f6e9 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:00.962487 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c38a673-a1f9-46f7-a99a-f97ffdcab0f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:00.963240 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c38a673-a1f9-46f7-a99a-f97ffdcab0f9 admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:00.963854 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c38a673-a1f9-46f7-a99a-f97ffdcab0f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:00.964433 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c38a673-a1f9-46f7-a99a-f97ffdcab0f9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:00.965218 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c38a673-a1f9-46f7-a99a-f97ffdcab0f9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:00.967853 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c38a673-a1f9-46f7-a99a-f97ffdcab0f9 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:00.976106 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:01.001397 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af6fa5cb-e1a2-44a6-9e9b-880281fdae7d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:01.002888 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af6fa5cb-e1a2-44a6-9e9b-880281fdae7d service cinder] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:01.018591 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 940/1904] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 00:57:00 2022] POST /identity/v3/projects => generated 383 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:57:01.019915 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c38a673-a1f9-46f7-a99a-f97ffdcab0f9 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:01.028570 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 965/1905] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:57:00 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 404 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 09 00:57:01.085752 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af6fa5cb-e1a2-44a6-9e9b-880281fdae7d service cinder] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:01.090288 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-af6fa5cb-e1a2-44a6-9e9b-880281fdae7d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:01.138222 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af6fa5cb-e1a2-44a6-9e9b-880281fdae7d service cinder] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:01.181896 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c38a673-a1f9-46f7-a99a-f97ffdcab0f9 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:01.191312 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b83017dd-77eb-4840-a3ba-011cb3809909 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:01.192097 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83017dd-77eb-4840-a3ba-011cb3809909 admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:01.192470 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83017dd-77eb-4840-a3ba-011cb3809909 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:01.192825 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83017dd-77eb-4840-a3ba-011cb3809909 admin admin] PATH_INFO: `/v3/users` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:01.195101 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b83017dd-77eb-4840-a3ba-011cb3809909 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:01.213017 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b83017dd-77eb-4840-a3ba-011cb3809909 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:01.244788 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 941/1906] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:57:01 2022] POST /identity/v3/users => generated 298 bytes in 223 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:57:01.244788 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b83017dd-77eb-4840-a3ba-011cb3809909 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:01.254454 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b116256-f083-447e-a16d-70961b1f27d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:01.255088 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b116256-f083-447e-a16d-70961b1f27d4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:01.255369 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b116256-f083-447e-a16d-70961b1f27d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:01.255675 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b116256-f083-447e-a16d-70961b1f27d4 admin admin] PATH_INFO: `/v3/projects/02c483e0e68343e0be05cac138769c2c` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:01.257304 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b116256-f083-447e-a16d-70961b1f27d4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=02c483e0e68343e0be05cac138769c2c)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:01.257787 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b83017dd-77eb-4840-a3ba-011cb3809909 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:01.259348 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b116256-f083-447e-a16d-70961b1f27d4 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:01.263407 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e29a5fbb-58a0-4c39-8567-ae784d9f2868 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:01.264156 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e29a5fbb-58a0-4c39-8567-ae784d9f2868 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:01.264432 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e29a5fbb-58a0-4c39-8567-ae784d9f2868 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:01.264632 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e29a5fbb-58a0-4c39-8567-ae784d9f2868 admin admin] PATH_INFO: `/v3/users/c8d21a27678b46d5aef210817abfd991` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:01.266151 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e29a5fbb-58a0-4c39-8567-ae784d9f2868 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c8d21a27678b46d5aef210817abfd991)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:01.269214 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e29a5fbb-58a0-4c39-8567-ae784d9f2868 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:01.301400 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:01.324996 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-8b116256-f083-447e-a16d-70961b1f27d4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '02c483e0e68343e0be05cac138769c2c', 'request_id': 'req-8b116256-f083-447e-a16d-70961b1f27d4'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:01.332972 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-8b116256-f083-447e-a16d-70961b1f27d4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '02c483e0e68343e0be05cac138769c2c', 'request_id': 'req-8b116256-f083-447e-a16d-70961b1f27d4'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:01.340471 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-8b116256-f083-447e-a16d-70961b1f27d4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '02c483e0e68343e0be05cac138769c2c', 'request_id': 'req-8b116256-f083-447e-a16d-70961b1f27d4'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:01.342756 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-e29a5fbb-58a0-4c39-8567-ae784d9f2868 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c8d21a27678b46d5aef210817abfd991'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:01.352694 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-e29a5fbb-58a0-4c39-8567-ae784d9f2868 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c8d21a27678b46d5aef210817abfd991'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:01.353429 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-e29a5fbb-58a0-4c39-8567-ae784d9f2868 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c8d21a27678b46d5aef210817abfd991'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:01.356294 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-8b116256-f083-447e-a16d-70961b1f27d4 admin admin] The token cache is being invalidate because project 02c483e0e68343e0be05cac138769c2c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 00:57:01.357243 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-8b116256-f083-447e-a16d-70961b1f27d4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:01.358420 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-8b116256-f083-447e-a16d-70961b1f27d4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:01.360465 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 966/1907] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 00:57:01 2022] DELETE /identity/v3/projects/02c483e0e68343e0be05cac138769c2c => generated 0 bytes in 276 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 09 00:57:01.361552 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b116256-f083-447e-a16d-70961b1f27d4 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:01.363192 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-8b116256-f083-447e-a16d-70961b1f27d4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:01.364075 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-e29a5fbb-58a0-4c39-8567-ae784d9f2868 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c8d21a27678b46d5aef210817abfd991'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:01.380532 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-e29a5fbb-58a0-4c39-8567-ae784d9f2868 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c8d21a27678b46d5aef210817abfd991'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:01.389595 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 942/1908] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 00:57:01 2022] DELETE /identity/v3/users/c8d21a27678b46d5aef210817abfd991 => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:01.394053 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e29a5fbb-58a0-4c39-8567-ae784d9f2868 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:01.395057 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-e29a5fbb-58a0-4c39-8567-ae784d9f2868 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:01.442069 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b116256-f083-447e-a16d-70961b1f27d4 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:01.452096 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=85,keystone:INSERT=5,keystone:DELETE=10 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:01.473420 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e29a5fbb-58a0-4c39-8567-ae784d9f2868 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:01.480164 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af47102e-66f4-42f3-b1d4-999740ce1532 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:01.481326 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af47102e-66f4-42f3-b1d4-999740ce1532 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:01.481614 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af47102e-66f4-42f3-b1d4-999740ce1532 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:01.482088 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af47102e-66f4-42f3-b1d4-999740ce1532 admin admin] PATH_INFO: `/v3/users/05a49698531a4efab9bc686b0415eb14` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:01.483290 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af47102e-66f4-42f3-b1d4-999740ce1532 admin admin] RBAC: Authorizing `identity:delete_user(user_id=05a49698531a4efab9bc686b0415eb14)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:01.485285 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af47102e-66f4-42f3-b1d4-999740ce1532 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:01.510501 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4c830ed-2d64-4141-83c5-84690c2cad3a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:01.510959 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4c830ed-2d64-4141-83c5-84690c2cad3a admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:01.511384 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4c830ed-2d64-4141-83c5-84690c2cad3a admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:01.511527 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4c830ed-2d64-4141-83c5-84690c2cad3a admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:01.512211 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4c830ed-2d64-4141-83c5-84690c2cad3a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:01.513896 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4c830ed-2d64-4141-83c5-84690c2cad3a admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:01.525445 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 967/1909] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:57:01 2022] GET /identity/v3/roles => generated 1420 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:57:01.526557 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2d2d4e-a590-4342-bf24-7f2dd6ed6027 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:01.526846 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2d2d4e-a590-4342-bf24-7f2dd6ed6027 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:01.527123 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2d2d4e-a590-4342-bf24-7f2dd6ed6027 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:01.545059 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-af47102e-66f4-42f3-b1d4-999740ce1532 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '05a49698531a4efab9bc686b0415eb14'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:01.547776 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-0b2d2d4e-a590-4342-bf24-7f2dd6ed6027 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:01.549325 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-0b2d2d4e-a590-4342-bf24-7f2dd6ed6027 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:01.554108 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-af47102e-66f4-42f3-b1d4-999740ce1532 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '05a49698531a4efab9bc686b0415eb14'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:01.554550 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-af47102e-66f4-42f3-b1d4-999740ce1532 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '05a49698531a4efab9bc686b0415eb14'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:01.557017 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 968/1910] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:57:01.560869 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b2d2d4e-a590-4342-bf24-7f2dd6ed6027 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:01.562451 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-af47102e-66f4-42f3-b1d4-999740ce1532 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '05a49698531a4efab9bc686b0415eb14'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:01.571558 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b2d2d4e-a590-4342-bf24-7f2dd6ed6027 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:01.577299 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df5b9fbd-a341-4ab8-907d-f719b34681d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:01.577959 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5b9fbd-a341-4ab8-907d-f719b34681d1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:01.578244 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5b9fbd-a341-4ab8-907d-f719b34681d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:01.578544 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5b9fbd-a341-4ab8-907d-f719b34681d1 admin admin] PATH_INFO: `/v3/projects/349ac70969174f8db0577dc4a138970e/users/facfaba0804b4ce4b3228911bc3747a2/roles/092036ee46ac427baf4d56468575f697` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:01.579331 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-af47102e-66f4-42f3-b1d4-999740ce1532 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '05a49698531a4efab9bc686b0415eb14'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:01.590505 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 943/1911] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 00:57:01 2022] DELETE /identity/v3/users/05a49698531a4efab9bc686b0415eb14 => generated 0 bytes in 195 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:01.591621 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af47102e-66f4-42f3-b1d4-999740ce1532 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:01.593276 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-af47102e-66f4-42f3-b1d4-999740ce1532 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:01.594585 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df5b9fbd-a341-4ab8-907d-f719b34681d1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=349ac70969174f8db0577dc4a138970e, user_id=facfaba0804b4ce4b3228911bc3747a2, role_id=092036ee46ac427baf4d56468575f697)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:01.597436 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df5b9fbd-a341-4ab8-907d-f719b34681d1 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:01.615534 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 969/1912] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:57:01 2022] PUT /identity/v3/projects/349ac70969174f8db0577dc4a138970e/users/facfaba0804b4ce4b3228911bc3747a2/roles/092036ee46ac427baf4d56468575f697 => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:01.617938 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df5b9fbd-a341-4ab8-907d-f719b34681d1 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:01.619686 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-df5b9fbd-a341-4ab8-907d-f719b34681d1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:01.660078 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=93,keystone:DELETE=5,keystone:INSERT=2 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:01.672066 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af47102e-66f4-42f3-b1d4-999740ce1532 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:01.696923 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df5b9fbd-a341-4ab8-907d-f719b34681d1 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:01.705045 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-550f0da4-c57f-467e-aaba-0f8058ba4192 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:01.706323 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-550f0da4-c57f-467e-aaba-0f8058ba4192 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:01.707263 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-550f0da4-c57f-467e-aaba-0f8058ba4192 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:01.707263 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-550f0da4-c57f-467e-aaba-0f8058ba4192 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:01.707865 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-550f0da4-c57f-467e-aaba-0f8058ba4192 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:01.710702 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-550f0da4-c57f-467e-aaba-0f8058ba4192 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:01.722815 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 970/1913] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:57:01 2022] GET /identity/v3/roles => generated 1420 bytes in 105 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:57:01.726468 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-550f0da4-c57f-467e-aaba-0f8058ba4192 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:01.744402 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-550f0da4-c57f-467e-aaba-0f8058ba4192 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:01.749422 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55504040-9106-4821-9b7c-593891c81ab0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:01.750122 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55504040-9106-4821-9b7c-593891c81ab0 service neutron] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:01.750471 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55504040-9106-4821-9b7c-593891c81ab0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:01.750816 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55504040-9106-4821-9b7c-593891c81ab0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:01.754787 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-55504040-9106-4821-9b7c-593891c81ab0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:01.757419 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08a21e41-d1b1-42f1-b372-b873322546fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:01.758227 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08a21e41-d1b1-42f1-b372-b873322546fa admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:01.758504 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08a21e41-d1b1-42f1-b372-b873322546fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:01.758776 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08a21e41-d1b1-42f1-b372-b873322546fa admin admin] PATH_INFO: `/v3/projects/349ac70969174f8db0577dc4a138970e/users/facfaba0804b4ce4b3228911bc3747a2/roles/092036ee46ac427baf4d56468575f697` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:01.761350 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08a21e41-d1b1-42f1-b372-b873322546fa admin admin] RBAC: Authorizing `identity:create_grant(project_id=349ac70969174f8db0577dc4a138970e, user_id=facfaba0804b4ce4b3228911bc3747a2, role_id=092036ee46ac427baf4d56468575f697)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:01.764327 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08a21e41-d1b1-42f1-b372-b873322546fa admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:01.770119 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55504040-9106-4821-9b7c-593891c81ab0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:01.771740 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55504040-9106-4821-9b7c-593891c81ab0 service neutron] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:01.786931 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 971/1914] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:57:01 2022] PUT /identity/v3/projects/349ac70969174f8db0577dc4a138970e/users/facfaba0804b4ce4b3228911bc3747a2/roles/092036ee46ac427baf4d56468575f697 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:01.788797 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af799c80-3383-41e2-80d6-3ed9f8c3b496 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:01.789332 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af799c80-3383-41e2-80d6-3ed9f8c3b496 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:01.789914 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af799c80-3383-41e2-80d6-3ed9f8c3b496 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:01.800214 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 944/1915] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:57:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 209 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:57:01.829424 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-af799c80-3383-41e2-80d6-3ed9f8c3b496 None None] MFA Rules not processed for user `facfaba0804b4ce4b3228911bc3747a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:01.832971 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-af799c80-3383-41e2-80d6-3ed9f8c3b496 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:01.946222 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 972/1916] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:01 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:57:01.958684 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55504040-9106-4821-9b7c-593891c81ab0 service neutron] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:01.960213 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-55504040-9106-4821-9b7c-593891c81ab0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:02.009639 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:02.030098 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af799c80-3383-41e2-80d6-3ed9f8c3b496 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:02.033061 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-af799c80-3383-41e2-80d6-3ed9f8c3b496 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:02.056177 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55504040-9106-4821-9b7c-593891c81ab0 service neutron] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:02.152640 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef44ed45-f900-4a82-bcf1-222f5297cc1d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3f48335668a4e8f934ac11ff66cd308', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:02.153570 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef44ed45-f900-4a82-bcf1-222f5297cc1d service cinder] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:02.153933 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef44ed45-f900-4a82-bcf1-222f5297cc1d service cinder] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:02.154305 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef44ed45-f900-4a82-bcf1-222f5297cc1d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:02.160368 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af799c80-3383-41e2-80d6-3ed9f8c3b496 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:02.166077 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef44ed45-f900-4a82-bcf1-222f5297cc1d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:02.166938 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f19bdf6-ac04-4b62-9b5f-c504f8b029e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:02.169106 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef44ed45-f900-4a82-bcf1-222f5297cc1d service cinder] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:02.181155 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 945/1917] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:57:01 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 223 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:57:02.183098 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f19bdf6-ac04-4b62-9b5f-c504f8b029e6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:02.184766 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f19bdf6-ac04-4b62-9b5f-c504f8b029e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:02.185212 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f19bdf6-ac04-4b62-9b5f-c504f8b029e6 admin admin] PATH_INFO: `/v3/projects/ad3618890d6848a781503ee33472cf7a` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:02.192044 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f19bdf6-ac04-4b62-9b5f-c504f8b029e6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ad3618890d6848a781503ee33472cf7a)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:02.194100 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f19bdf6-ac04-4b62-9b5f-c504f8b029e6 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:02.284573 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=96,keystone:INSERT=1,keystone:DELETE=3 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:02.292343 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-5f19bdf6-ac04-4b62-9b5f-c504f8b029e6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ad3618890d6848a781503ee33472cf7a', 'request_id': 'req-5f19bdf6-ac04-4b62-9b5f-c504f8b029e6'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:02.307925 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-5f19bdf6-ac04-4b62-9b5f-c504f8b029e6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ad3618890d6848a781503ee33472cf7a', 'request_id': 'req-5f19bdf6-ac04-4b62-9b5f-c504f8b029e6'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:02.316303 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-5f19bdf6-ac04-4b62-9b5f-c504f8b029e6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ad3618890d6848a781503ee33472cf7a', 'request_id': 'req-5f19bdf6-ac04-4b62-9b5f-c504f8b029e6'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:02.326885 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef44ed45-f900-4a82-bcf1-222f5297cc1d service cinder] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:02.328063 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-ef44ed45-f900-4a82-bcf1-222f5297cc1d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:02.340157 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-5f19bdf6-ac04-4b62-9b5f-c504f8b029e6 admin admin] The token cache is being invalidate because project ad3618890d6848a781503ee33472cf7a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 00:57:02.340707 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-5f19bdf6-ac04-4b62-9b5f-c504f8b029e6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:02.341531 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-5f19bdf6-ac04-4b62-9b5f-c504f8b029e6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:02.343638 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 973/1918] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 00:57:02 2022] DELETE /identity/v3/projects/ad3618890d6848a781503ee33472cf7a => generated 0 bytes in 314 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 09 00:57:02.344139 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef44ed45-f900-4a82-bcf1-222f5297cc1d service cinder] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:02.348953 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e5860e4-b674-4766-9bb6-fd2aafd5ee48 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:02.349372 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e5860e4-b674-4766-9bb6-fd2aafd5ee48 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:02.349752 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e5860e4-b674-4766-9bb6-fd2aafd5ee48 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:02.383312 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-0e5860e4-b674-4766-9bb6-fd2aafd5ee48 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:02.384696 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-0e5860e4-b674-4766-9bb6-fd2aafd5ee48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:02.413488 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aac076d-cb2f-4343-b1fb-2b6015bba497 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:02.414131 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aac076d-cb2f-4343-b1fb-2b6015bba497 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:02.414412 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aac076d-cb2f-4343-b1fb-2b6015bba497 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:02.414681 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aac076d-cb2f-4343-b1fb-2b6015bba497 admin admin] PATH_INFO: `/v3/projects/bef1dcb3d43f44de8a3e058b4dee56ab` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:02.416150 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aac076d-cb2f-4343-b1fb-2b6015bba497 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bef1dcb3d43f44de8a3e058b4dee56ab)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:02.418087 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aac076d-cb2f-4343-b1fb-2b6015bba497 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:02.482046 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 974/1919] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:57:02.482046 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e5860e4-b674-4766-9bb6-fd2aafd5ee48 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:02.482046 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-0e5860e4-b674-4766-9bb6-fd2aafd5ee48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:02.498173 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-4aac076d-cb2f-4343-b1fb-2b6015bba497 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bef1dcb3d43f44de8a3e058b4dee56ab', 'request_id': 'req-4aac076d-cb2f-4343-b1fb-2b6015bba497'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:02.508896 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-4aac076d-cb2f-4343-b1fb-2b6015bba497 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bef1dcb3d43f44de8a3e058b4dee56ab', 'request_id': 'req-4aac076d-cb2f-4343-b1fb-2b6015bba497'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:02.518790 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-4aac076d-cb2f-4343-b1fb-2b6015bba497 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bef1dcb3d43f44de8a3e058b4dee56ab', 'request_id': 'req-4aac076d-cb2f-4343-b1fb-2b6015bba497'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:02.537540 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-4aac076d-cb2f-4343-b1fb-2b6015bba497 admin admin] The token cache is being invalidate because project bef1dcb3d43f44de8a3e058b4dee56ab was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81453) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 00:57:02.538408 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-4aac076d-cb2f-4343-b1fb-2b6015bba497 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:02.539693 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-4aac076d-cb2f-4343-b1fb-2b6015bba497 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:02.544591 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 946/1920] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 00:57:02 2022] DELETE /identity/v3/projects/bef1dcb3d43f44de8a3e058b4dee56ab => generated 0 bytes in 216 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 09 00:57:02.546295 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5fb2dca-019c-4d1d-88dc-fb1b620156a1 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:02.546740 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5fb2dca-019c-4d1d-88dc-fb1b620156a1 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:02.547800 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5fb2dca-019c-4d1d-88dc-fb1b620156a1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:02.553617 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e5860e4-b674-4766-9bb6-fd2aafd5ee48 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:02.572383 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-a5fb2dca-019c-4d1d-88dc-fb1b620156a1 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:02.573790 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-a5fb2dca-019c-4d1d-88dc-fb1b620156a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:02.585550 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:02.612309 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17e217b9-1802-4f28-9710-5dfda9136981 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3f48335668a4e8f934ac11ff66cd308', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:02.613372 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e217b9-1802-4f28-9710-5dfda9136981 service cinder] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:02.613750 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e217b9-1802-4f28-9710-5dfda9136981 service cinder] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:02.613910 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e217b9-1802-4f28-9710-5dfda9136981 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:02.616471 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-17e217b9-1802-4f28-9710-5dfda9136981 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:02.653839 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 947/1921] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:57:02.653839 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5fb2dca-019c-4d1d-88dc-fb1b620156a1 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:02.653839 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-a5fb2dca-019c-4d1d-88dc-fb1b620156a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:02.665737 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5fb2dca-019c-4d1d-88dc-fb1b620156a1 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:02.668712 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=95,keystone:DELETE=4,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:02.671465 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b28ab8ba-1d95-4595-bb5e-138e7ad2185a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:02.672284 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b28ab8ba-1d95-4595-bb5e-138e7ad2185a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:02.673498 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b28ab8ba-1d95-4595-bb5e-138e7ad2185a admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:02.673498 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b28ab8ba-1d95-4595-bb5e-138e7ad2185a admin admin] PATH_INFO: `/v3/users/647c3bb7f2fe44d39b6bdfed4a75a61a` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:02.674212 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b28ab8ba-1d95-4595-bb5e-138e7ad2185a admin admin] RBAC: Authorizing `identity:delete_user(user_id=647c3bb7f2fe44d39b6bdfed4a75a61a)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:02.676571 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b28ab8ba-1d95-4595-bb5e-138e7ad2185a admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:02.688388 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17e217b9-1802-4f28-9710-5dfda9136981 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:02.690836 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17e217b9-1802-4f28-9710-5dfda9136981 service cinder] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:02.700334 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 975/1922] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:57:02 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 223 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:57:02.701002 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17e217b9-1802-4f28-9710-5dfda9136981 service cinder] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:02.714192 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17e217b9-1802-4f28-9710-5dfda9136981 service cinder] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:02.720287 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-592de9d3-5619-4fb4-aed7-32eb890bec1f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:02.721060 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-592de9d3-5619-4fb4-aed7-32eb890bec1f admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:02.721427 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-592de9d3-5619-4fb4-aed7-32eb890bec1f admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:02.721843 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-592de9d3-5619-4fb4-aed7-32eb890bec1f admin admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:02.722749 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-592de9d3-5619-4fb4-aed7-32eb890bec1f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:57:02.723118 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-592de9d3-5619-4fb4-aed7-32eb890bec1f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:02.724981 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-592de9d3-5619-4fb4-aed7-32eb890bec1f admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:02.738515 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 976/1923] 10.222.0.33 () {58 vars in 1153 bytes} [Sun Oct 9 00:57:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:57:02.742669 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-592de9d3-5619-4fb4-aed7-32eb890bec1f admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:02.744269 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-b28ab8ba-1d95-4595-bb5e-138e7ad2185a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '647c3bb7f2fe44d39b6bdfed4a75a61a'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:02.761880 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-592de9d3-5619-4fb4-aed7-32eb890bec1f admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:02.762168 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-b28ab8ba-1d95-4595-bb5e-138e7ad2185a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '647c3bb7f2fe44d39b6bdfed4a75a61a'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:02.763249 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-b28ab8ba-1d95-4595-bb5e-138e7ad2185a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '647c3bb7f2fe44d39b6bdfed4a75a61a'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:02.767549 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-435dd067-a32f-4198-9871-7adc49975cb2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:02.768239 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-435dd067-a32f-4198-9871-7adc49975cb2 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:02.768520 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-435dd067-a32f-4198-9871-7adc49975cb2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:02.768879 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-435dd067-a32f-4198-9871-7adc49975cb2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:02.769662 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-435dd067-a32f-4198-9871-7adc49975cb2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:02.771968 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-435dd067-a32f-4198-9871-7adc49975cb2 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:02.773954 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-b28ab8ba-1d95-4595-bb5e-138e7ad2185a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '647c3bb7f2fe44d39b6bdfed4a75a61a'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:02.792206 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-b28ab8ba-1d95-4595-bb5e-138e7ad2185a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '647c3bb7f2fe44d39b6bdfed4a75a61a'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:02.801072 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 948/1924] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 00:57:02 2022] DELETE /identity/v3/users/647c3bb7f2fe44d39b6bdfed4a75a61a => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:02.807345 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 977/1925] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 00:57:02 2022] POST /identity/v3/projects => generated 385 bytes in 65 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:57:02.811373 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b28ab8ba-1d95-4595-bb5e-138e7ad2185a admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:02.813253 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-b28ab8ba-1d95-4595-bb5e-138e7ad2185a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:02.913159 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b28ab8ba-1d95-4595-bb5e-138e7ad2185a admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:02.962582 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:03.018289 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7776f94-e62f-4715-8de0-d1afca149dfc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:03.018923 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7776f94-e62f-4715-8de0-d1afca149dfc admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:03.019205 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7776f94-e62f-4715-8de0-d1afca149dfc admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:03.031817 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7776f94-e62f-4715-8de0-d1afca149dfc admin admin] PATH_INFO: `/v3/users` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:03.032720 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7776f94-e62f-4715-8de0-d1afca149dfc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:03.034875 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7776f94-e62f-4715-8de0-d1afca149dfc admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:03.068891 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 949/1926] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:57:02 2022] POST /identity/v3/users => generated 293 bytes in 258 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:57:03.078193 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-435dd067-a32f-4198-9871-7adc49975cb2 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:03.089066 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-435dd067-a32f-4198-9871-7adc49975cb2 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:03.098929 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d23f4e1-c0f8-4671-8f36-d8b2d3b65c13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:03.099346 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d23f4e1-c0f8-4671-8f36-d8b2d3b65c13 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:03.099676 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d23f4e1-c0f8-4671-8f36-d8b2d3b65c13 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:03.099957 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d23f4e1-c0f8-4671-8f36-d8b2d3b65c13 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:03.100606 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d23f4e1-c0f8-4671-8f36-d8b2d3b65c13 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:03.102433 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d23f4e1-c0f8-4671-8f36-d8b2d3b65c13 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:03.145474 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 978/1927] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:57:03 2022] GET /identity/v3/roles => generated 1420 bytes in 73 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:57:03.149716 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7776f94-e62f-4715-8de0-d1afca149dfc admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:03.161835 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7776f94-e62f-4715-8de0-d1afca149dfc admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:03.165707 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcb8778c-8ce8-44be-88d1-be38980dd3e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:03.166393 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcb8778c-8ce8-44be-88d1-be38980dd3e3 admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:03.166647 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcb8778c-8ce8-44be-88d1-be38980dd3e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:03.166928 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcb8778c-8ce8-44be-88d1-be38980dd3e3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:03.167586 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcb8778c-8ce8-44be-88d1-be38980dd3e3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:03.169217 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcb8778c-8ce8-44be-88d1-be38980dd3e3 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:03.171134 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d23f4e1-c0f8-4671-8f36-d8b2d3b65c13 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:03.173500 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-4d23f4e1-c0f8-4671-8f36-d8b2d3b65c13 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:03.192157 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d23f4e1-c0f8-4671-8f36-d8b2d3b65c13 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:03.197012 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9aae214-6d56-4969-b255-e25f22c86278 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:03.198033 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9aae214-6d56-4969-b255-e25f22c86278 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:03.198315 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9aae214-6d56-4969-b255-e25f22c86278 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:03.198585 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9aae214-6d56-4969-b255-e25f22c86278 admin admin] PATH_INFO: `/v3/projects/d4df28446e714a29b69ebc706ea6be93` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:03.199857 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9aae214-6d56-4969-b255-e25f22c86278 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d4df28446e714a29b69ebc706ea6be93)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:03.201827 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9aae214-6d56-4969-b255-e25f22c86278 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:03.203311 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 950/1928] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:57:03 2022] GET /identity/v3/roles => generated 1420 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:57:03.204291 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcb8778c-8ce8-44be-88d1-be38980dd3e3 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:03.215357 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcb8778c-8ce8-44be-88d1-be38980dd3e3 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:03.224915 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f8cbfdf-f2a8-4bdc-846d-bd5bc1612e44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:03.225593 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8cbfdf-f2a8-4bdc-846d-bd5bc1612e44 admin admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:03.225880 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8cbfdf-f2a8-4bdc-846d-bd5bc1612e44 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:03.226159 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f8cbfdf-f2a8-4bdc-846d-bd5bc1612e44 admin admin] PATH_INFO: `/v3/projects/619223fd7bd04d9e9d9f1a866bd1e2aa/users/3f85d2ae83134254aa1fb11c1e500767/roles/588016933bab436d8b7c3ac76827356d` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:03.246000 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f8cbfdf-f2a8-4bdc-846d-bd5bc1612e44 admin admin] RBAC: Authorizing `identity:create_grant(project_id=619223fd7bd04d9e9d9f1a866bd1e2aa, user_id=3f85d2ae83134254aa1fb11c1e500767, role_id=588016933bab436d8b7c3ac76827356d)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:03.248835 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f8cbfdf-f2a8-4bdc-846d-bd5bc1612e44 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:03.270202 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 951/1929] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:57:03 2022] PUT /identity/v3/projects/619223fd7bd04d9e9d9f1a866bd1e2aa/users/3f85d2ae83134254aa1fb11c1e500767/roles/588016933bab436d8b7c3ac76827356d => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:03.273981 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-a9aae214-6d56-4969-b255-e25f22c86278 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd4df28446e714a29b69ebc706ea6be93', 'request_id': 'req-a9aae214-6d56-4969-b255-e25f22c86278'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:03.280453 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-a9aae214-6d56-4969-b255-e25f22c86278 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd4df28446e714a29b69ebc706ea6be93', 'request_id': 'req-a9aae214-6d56-4969-b255-e25f22c86278'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:03.281598 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce39fc0b-7b2f-44a3-8d10-de014e48cf9d None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:03.282104 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce39fc0b-7b2f-44a3-8d10-de014e48cf9d None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:03.282383 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce39fc0b-7b2f-44a3-8d10-de014e48cf9d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:03.289754 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-a9aae214-6d56-4969-b255-e25f22c86278 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd4df28446e714a29b69ebc706ea6be93', 'request_id': 'req-a9aae214-6d56-4969-b255-e25f22c86278'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:03.322337 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-a9aae214-6d56-4969-b255-e25f22c86278 admin admin] The token cache is being invalidate because project d4df28446e714a29b69ebc706ea6be93 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 00:57:03.322626 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-a9aae214-6d56-4969-b255-e25f22c86278 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:03.328456 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-a9aae214-6d56-4969-b255-e25f22c86278 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:03.332404 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-ce39fc0b-7b2f-44a3-8d10-de014e48cf9d None None] MFA Rules not processed for user `3f85d2ae83134254aa1fb11c1e500767`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:03.334923 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-ce39fc0b-7b2f-44a3-8d10-de014e48cf9d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:03.336331 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 979/1930] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 00:57:03 2022] DELETE /identity/v3/projects/d4df28446e714a29b69ebc706ea6be93 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 09 00:57:03.338543 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff4ff7ee-37cf-4ac1-9d0b-fedf051edf09 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:03.338834 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff4ff7ee-37cf-4ac1-9d0b-fedf051edf09 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:03.339110 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff4ff7ee-37cf-4ac1-9d0b-fedf051edf09 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:03.380972 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-ff4ff7ee-37cf-4ac1-9d0b-fedf051edf09 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:03.382592 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-ff4ff7ee-37cf-4ac1-9d0b-fedf051edf09 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:03.430146 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 952/1931] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:03 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:57:03.434438 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce39fc0b-7b2f-44a3-8d10-de014e48cf9d None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:03.435911 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-ce39fc0b-7b2f-44a3-8d10-de014e48cf9d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:03.469451 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 980/1932] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:57:03.476731 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d297e9-1796-4a53-b228-9cf54f11fca2 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:03.477127 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d297e9-1796-4a53-b228-9cf54f11fca2 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:03.477543 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d297e9-1796-4a53-b228-9cf54f11fca2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:03.497176 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-14d297e9-1796-4a53-b228-9cf54f11fca2 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:03.498376 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-14d297e9-1796-4a53-b228-9cf54f11fca2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:03.508214 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 981/1933] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 09 00:57:03.512326 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14d297e9-1796-4a53-b228-9cf54f11fca2 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:03.513003 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce39fc0b-7b2f-44a3-8d10-de014e48cf9d None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:03.517928 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-031f4c2e-a4a2-4c3f-968f-f346eeb48ecb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:03.518573 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-031f4c2e-a4a2-4c3f-968f-f346eeb48ecb admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:03.518861 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-031f4c2e-a4a2-4c3f-968f-f346eeb48ecb admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:03.519138 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-031f4c2e-a4a2-4c3f-968f-f346eeb48ecb admin admin] PATH_INFO: `/v3/projects` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:03.519878 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-031f4c2e-a4a2-4c3f-968f-f346eeb48ecb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:03.521720 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14d297e9-1796-4a53-b228-9cf54f11fca2 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:03.521851 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-031f4c2e-a4a2-4c3f-968f-f346eeb48ecb admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:03.526323 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8725c7c2-9c7e-4fa8-9f99-442071cb5a08 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:03.526907 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8725c7c2-9c7e-4fa8-9f99-442071cb5a08 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:03.527356 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8725c7c2-9c7e-4fa8-9f99-442071cb5a08 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:03.527460 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8725c7c2-9c7e-4fa8-9f99-442071cb5a08 admin admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:03.528118 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8725c7c2-9c7e-4fa8-9f99-442071cb5a08 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:57:03.528419 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8725c7c2-9c7e-4fa8-9f99-442071cb5a08 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:03.529992 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8725c7c2-9c7e-4fa8-9f99-442071cb5a08 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:03.539130 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:03.543245 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 982/1934] 10.222.0.33 () {58 vars in 1153 bytes} [Sun Oct 9 00:57:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:57:03.547129 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8725c7c2-9c7e-4fa8-9f99-442071cb5a08 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:03.554676 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 953/1935] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 00:57:03 2022] POST /identity/v3/projects => generated 387 bytes in 117 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:57:03.555769 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-031f4c2e-a4a2-4c3f-968f-f346eeb48ecb admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:03.558120 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8725c7c2-9c7e-4fa8-9f99-442071cb5a08 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:03.560066 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:03.618860 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e30534aa-6aed-4258-afcf-f5eeaea0b8e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:03.620184 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e30534aa-6aed-4258-afcf-f5eeaea0b8e7 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:03.620592 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e30534aa-6aed-4258-afcf-f5eeaea0b8e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:03.620901 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-031f4c2e-a4a2-4c3f-968f-f346eeb48ecb admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:03.621029 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e30534aa-6aed-4258-afcf-f5eeaea0b8e7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:03.621774 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e30534aa-6aed-4258-afcf-f5eeaea0b8e7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:03.623914 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e30534aa-6aed-4258-afcf-f5eeaea0b8e7 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:03.625556 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2659d56-4205-4310-a758-914a4195b746 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:03.626306 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2659d56-4205-4310-a758-914a4195b746 admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:03.626592 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2659d56-4205-4310-a758-914a4195b746 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:03.627542 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2659d56-4205-4310-a758-914a4195b746 admin admin] PATH_INFO: `/v3/users` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:03.628738 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2659d56-4205-4310-a758-914a4195b746 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:03.631190 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2659d56-4205-4310-a758-914a4195b746 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:03.655638 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 983/1936] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 00:57:03 2022] POST /identity/v3/projects => generated 379 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 09 00:57:03.659308 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 954/1937] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:57:03 2022] POST /identity/v3/users => generated 294 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 09 00:57:03.660498 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2659d56-4205-4310-a758-914a4195b746 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:03.660668 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e30534aa-6aed-4258-afcf-f5eeaea0b8e7 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:03.672158 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2659d56-4205-4310-a758-914a4195b746 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:03.734904 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e30534aa-6aed-4258-afcf-f5eeaea0b8e7 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:03.742600 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c4905a-c847-4cf4-b6b4-0a68dad21244 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:03.743380 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c4905a-c847-4cf4-b6b4-0a68dad21244 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:03.743769 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c4905a-c847-4cf4-b6b4-0a68dad21244 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:03.744118 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c4905a-c847-4cf4-b6b4-0a68dad21244 admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:03.744828 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97c4905a-c847-4cf4-b6b4-0a68dad21244 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:03.745708 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c142effd-b46a-488b-9d97-d8c35d382692 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:03.746432 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c142effd-b46a-488b-9d97-d8c35d382692 admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:03.746758 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c142effd-b46a-488b-9d97-d8c35d382692 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:03.747101 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c142effd-b46a-488b-9d97-d8c35d382692 admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:03.747262 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97c4905a-c847-4cf4-b6b4-0a68dad21244 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:03.747967 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c142effd-b46a-488b-9d97-d8c35d382692 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:03.749875 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c142effd-b46a-488b-9d97-d8c35d382692 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:03.761127 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 955/1938] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:57:03 2022] GET /identity/v3/roles => generated 1420 bytes in 101 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:57:03.765103 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c142effd-b46a-488b-9d97-d8c35d382692 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:03.779339 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c142effd-b46a-488b-9d97-d8c35d382692 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:03.779467 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 984/1939] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:57:03 2022] POST /identity/v3/users => generated 290 bytes in 119 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:57:03.782988 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c4905a-c847-4cf4-b6b4-0a68dad21244 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:03.784125 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb76e3e5-c70c-44a9-ba96-4cf8a54eca22 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:03.784716 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb76e3e5-c70c-44a9-ba96-4cf8a54eca22 admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:03.784995 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb76e3e5-c70c-44a9-ba96-4cf8a54eca22 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:03.785281 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb76e3e5-c70c-44a9-ba96-4cf8a54eca22 admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:03.785890 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb76e3e5-c70c-44a9-ba96-4cf8a54eca22 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:03.787462 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb76e3e5-c70c-44a9-ba96-4cf8a54eca22 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:03.793574 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c4905a-c847-4cf4-b6b4-0a68dad21244 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:03.798181 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-385a7cdb-a741-48c0-acb2-7b904382256f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:03.798770 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-385a7cdb-a741-48c0-acb2-7b904382256f admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:03.799050 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-385a7cdb-a741-48c0-acb2-7b904382256f admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:03.799138 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 956/1940] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:57:03 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 09 00:57:03.799892 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-385a7cdb-a741-48c0-acb2-7b904382256f admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:03.800659 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-385a7cdb-a741-48c0-acb2-7b904382256f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:03.802280 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-385a7cdb-a741-48c0-acb2-7b904382256f admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:03.804108 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb76e3e5-c70c-44a9-ba96-4cf8a54eca22 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:03.815876 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 985/1941] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:57:03 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 09 00:57:03.816026 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb76e3e5-c70c-44a9-ba96-4cf8a54eca22 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:03.817214 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-385a7cdb-a741-48c0-acb2-7b904382256f admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:03.819889 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-435629b2-2f54-4934-b251-0187964610dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:03.820533 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-435629b2-2f54-4934-b251-0187964610dc admin admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:03.820810 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-435629b2-2f54-4934-b251-0187964610dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:03.821081 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-435629b2-2f54-4934-b251-0187964610dc admin admin] PATH_INFO: `/v3/projects/dcc60c20223e4b8892d82f207177f9cc/users/fdfcc4b4521743fcb7b4a804fe229322/roles/588016933bab436d8b7c3ac76827356d` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:03.826583 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-385a7cdb-a741-48c0-acb2-7b904382256f admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:03.831217 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be303093-0e32-4536-82a0-c805e11bdac3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:03.831823 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be303093-0e32-4536-82a0-c805e11bdac3 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:03.832114 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be303093-0e32-4536-82a0-c805e11bdac3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:03.832385 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be303093-0e32-4536-82a0-c805e11bdac3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:03.833009 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be303093-0e32-4536-82a0-c805e11bdac3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:03.834587 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be303093-0e32-4536-82a0-c805e11bdac3 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:03.835557 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-435629b2-2f54-4934-b251-0187964610dc admin admin] RBAC: Authorizing `identity:create_grant(project_id=dcc60c20223e4b8892d82f207177f9cc, user_id=fdfcc4b4521743fcb7b4a804fe229322, role_id=588016933bab436d8b7c3ac76827356d)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:03.838218 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-435629b2-2f54-4934-b251-0187964610dc admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:03.846990 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 986/1942] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:57:03 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 09 00:57:03.851154 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be303093-0e32-4536-82a0-c805e11bdac3 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:03.852579 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 957/1943] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:57:03 2022] PUT /identity/v3/projects/dcc60c20223e4b8892d82f207177f9cc/users/fdfcc4b4521743fcb7b4a804fe229322/roles/588016933bab436d8b7c3ac76827356d => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:03.861829 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be303093-0e32-4536-82a0-c805e11bdac3 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:03.864414 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f140370c-ca58-4e55-8985-f2a4558d8fcc None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:03.864774 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f140370c-ca58-4e55-8985-f2a4558d8fcc None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:03.865045 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f140370c-ca58-4e55-8985-f2a4558d8fcc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:03.892435 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-f140370c-ca58-4e55-8985-f2a4558d8fcc None None] MFA Rules not processed for user `fdfcc4b4521743fcb7b4a804fe229322`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:03.893287 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-f140370c-ca58-4e55-8985-f2a4558d8fcc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:03.910562 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:03.921707 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08ed0690-06ff-42ce-905f-a8e4503866d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:03.922370 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ed0690-06ff-42ce-905f-a8e4503866d2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:03.922652 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ed0690-06ff-42ce-905f-a8e4503866d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:03.922925 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ed0690-06ff-42ce-905f-a8e4503866d2 admin admin] PATH_INFO: `/v3/projects/c6be54a4bceb4bc8b03ba71439906556/users/b536979b218641bab926bb51c84789e0/roles/588016933bab436d8b7c3ac76827356d` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:03.935861 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:03.938884 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08ed0690-06ff-42ce-905f-a8e4503866d2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c6be54a4bceb4bc8b03ba71439906556, user_id=b536979b218641bab926bb51c84789e0, role_id=588016933bab436d8b7c3ac76827356d)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:03.942112 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08ed0690-06ff-42ce-905f-a8e4503866d2 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:03.955323 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 987/1944] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:57:03 2022] PUT /identity/v3/projects/c6be54a4bceb4bc8b03ba71439906556/users/b536979b218641bab926bb51c84789e0/roles/588016933bab436d8b7c3ac76827356d => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 09 00:57:03.959576 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 958/1945] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:03 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 09 00:57:03.963895 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08ed0690-06ff-42ce-905f-a8e4503866d2 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:03.966357 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea52c916-71b1-4a74-a6f1-ee42609ab1f8 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:03.966647 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea52c916-71b1-4a74-a6f1-ee42609ab1f8 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:03.966922 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea52c916-71b1-4a74-a6f1-ee42609ab1f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:03.995679 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-ea52c916-71b1-4a74-a6f1-ee42609ab1f8 None None] MFA Rules not processed for user `b536979b218641bab926bb51c84789e0`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:03.996961 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-ea52c916-71b1-4a74-a6f1-ee42609ab1f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:04.027098 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08ed0690-06ff-42ce-905f-a8e4503866d2 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:04.032190 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76aa981d-e730-4bc3-93cb-2d35238b542a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:04.032790 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76aa981d-e730-4bc3-93cb-2d35238b542a admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:04.033072 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76aa981d-e730-4bc3-93cb-2d35238b542a admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:04.033347 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76aa981d-e730-4bc3-93cb-2d35238b542a admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:04.034621 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76aa981d-e730-4bc3-93cb-2d35238b542a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:04.036630 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76aa981d-e730-4bc3-93cb-2d35238b542a admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:04.063809 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 988/1946] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 00:57:03 2022] POST /identity/v3/projects => generated 385 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 09 00:57:04.064318 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 959/1947] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:03 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:57:04.068585 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea52c916-71b1-4a74-a6f1-ee42609ab1f8 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:04.068710 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76aa981d-e730-4bc3-93cb-2d35238b542a admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:04.080093 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea52c916-71b1-4a74-a6f1-ee42609ab1f8 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:04.132921 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76aa981d-e730-4bc3-93cb-2d35238b542a admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:04.137506 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-216ffec3-bb54-4344-a67f-476a6002a21e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:04.138108 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-216ffec3-bb54-4344-a67f-476a6002a21e admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:04.138388 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-216ffec3-bb54-4344-a67f-476a6002a21e admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:04.138657 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-216ffec3-bb54-4344-a67f-476a6002a21e admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:04.139282 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-216ffec3-bb54-4344-a67f-476a6002a21e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:04.140400 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6fd6a72-cb79-44ad-a276-94912a2fb51d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:04.141016 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6fd6a72-cb79-44ad-a276-94912a2fb51d admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:04.141205 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-216ffec3-bb54-4344-a67f-476a6002a21e admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:04.141299 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6fd6a72-cb79-44ad-a276-94912a2fb51d admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:04.141556 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6fd6a72-cb79-44ad-a276-94912a2fb51d admin admin] PATH_INFO: `/v3/users` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:04.142181 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6fd6a72-cb79-44ad-a276-94912a2fb51d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:04.144945 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6fd6a72-cb79-44ad-a276-94912a2fb51d admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:04.169444 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 960/1948] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:57:04 2022] POST /identity/v3/users => generated 299 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:57:04.172830 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 989/1949] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 00:57:04 2022] POST /identity/v3/projects => generated 379 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 09 00:57:04.173213 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6fd6a72-cb79-44ad-a276-94912a2fb51d admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:04.176627 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-216ffec3-bb54-4344-a67f-476a6002a21e admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:04.183426 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6fd6a72-cb79-44ad-a276-94912a2fb51d admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:04.228382 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:04.245303 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-216ffec3-bb54-4344-a67f-476a6002a21e admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:04.245499 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bbee5e7-e631-4c20-b583-d744cf4e9fe4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:04.246112 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bbee5e7-e631-4c20-b583-d744cf4e9fe4 admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:04.246393 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bbee5e7-e631-4c20-b583-d744cf4e9fe4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:04.246663 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bbee5e7-e631-4c20-b583-d744cf4e9fe4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:04.247299 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bbee5e7-e631-4c20-b583-d744cf4e9fe4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:04.248920 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bbee5e7-e631-4c20-b583-d744cf4e9fe4 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:04.250193 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c35bfe07-1fed-4410-9552-6452b6a54c4e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:04.250794 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c35bfe07-1fed-4410-9552-6452b6a54c4e admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:04.251077 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c35bfe07-1fed-4410-9552-6452b6a54c4e admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:04.251351 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c35bfe07-1fed-4410-9552-6452b6a54c4e admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:04.251774 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:04.254053 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c35bfe07-1fed-4410-9552-6452b6a54c4e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:04.254505 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c35bfe07-1fed-4410-9552-6452b6a54c4e admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:04.265019 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 961/1950] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:57:04 2022] GET /identity/v3/roles => generated 1420 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 09 00:57:04.266125 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bbee5e7-e631-4c20-b583-d744cf4e9fe4 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:04.282793 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 990/1951] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:57:04 2022] POST /identity/v3/users => generated 296 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:57:04.286695 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c35bfe07-1fed-4410-9552-6452b6a54c4e admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:04.290049 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bbee5e7-e631-4c20-b583-d744cf4e9fe4 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:04.294930 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ecb19ae-7a3f-4d03-b206-b6dd36df7c76 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:04.295613 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ecb19ae-7a3f-4d03-b206-b6dd36df7c76 admin admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:04.295905 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ecb19ae-7a3f-4d03-b206-b6dd36df7c76 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:04.296226 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ecb19ae-7a3f-4d03-b206-b6dd36df7c76 admin admin] PATH_INFO: `/v3/projects/aad4ea28111941a19ea89bf076cc9100/users/5b2c6b9740a14ee4bfeac6e3c45eec2f/roles/092036ee46ac427baf4d56468575f697` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:04.297841 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c35bfe07-1fed-4410-9552-6452b6a54c4e admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:04.307556 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d939a9d-bfdb-4764-93d4-2755518805da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:04.308381 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d939a9d-bfdb-4764-93d4-2755518805da admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:04.308858 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d939a9d-bfdb-4764-93d4-2755518805da admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:04.309281 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d939a9d-bfdb-4764-93d4-2755518805da admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:04.310075 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d939a9d-bfdb-4764-93d4-2755518805da admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:04.312074 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ecb19ae-7a3f-4d03-b206-b6dd36df7c76 admin admin] RBAC: Authorizing `identity:create_grant(project_id=aad4ea28111941a19ea89bf076cc9100, user_id=5b2c6b9740a14ee4bfeac6e3c45eec2f, role_id=092036ee46ac427baf4d56468575f697)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:04.312345 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d939a9d-bfdb-4764-93d4-2755518805da admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:04.315883 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ecb19ae-7a3f-4d03-b206-b6dd36df7c76 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:04.326070 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 991/1952] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:57:04 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:57:04.330058 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d939a9d-bfdb-4764-93d4-2755518805da admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:04.332462 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 962/1953] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:57:04 2022] PUT /identity/v3/projects/aad4ea28111941a19ea89bf076cc9100/users/5b2c6b9740a14ee4bfeac6e3c45eec2f/roles/092036ee46ac427baf4d56468575f697 => generated 0 bytes in 67 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:04.336246 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ecb19ae-7a3f-4d03-b206-b6dd36df7c76 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:04.340563 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d939a9d-bfdb-4764-93d4-2755518805da admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:04.436191 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ecb19ae-7a3f-4d03-b206-b6dd36df7c76 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:04.446318 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d4a90a-35eb-421e-a9ea-0396502be365 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:04.446934 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d4a90a-35eb-421e-a9ea-0396502be365 admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:04.450934 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d4a90a-35eb-421e-a9ea-0396502be365 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:04.451574 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d4a90a-35eb-421e-a9ea-0396502be365 admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:04.451970 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35d4a90a-35eb-421e-a9ea-0396502be365 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:04.453830 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35d4a90a-35eb-421e-a9ea-0396502be365 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:04.477832 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0eecc02-ab70-4d87-be08-c6a91d90707f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:04.478699 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0eecc02-ab70-4d87-be08-c6a91d90707f admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:04.478987 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0eecc02-ab70-4d87-be08-c6a91d90707f admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:04.479263 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0eecc02-ab70-4d87-be08-c6a91d90707f admin admin] PATH_INFO: `/v3/projects/d8abf6d39eac40c28c62cd3a8a7d250f/users/3b2a5747ed0443e1928b637152e4bfdf/roles/092036ee46ac427baf4d56468575f697` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:04.483678 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 963/1954] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:57:04 2022] GET /identity/v3/roles => generated 1420 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 09 00:57:04.488404 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d4a90a-35eb-421e-a9ea-0396502be365 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:04.504974 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d4a90a-35eb-421e-a9ea-0396502be365 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:04.510034 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b879decc-0c57-43b9-a34f-9c56c7ebc036 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:04.510815 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b879decc-0c57-43b9-a34f-9c56c7ebc036 admin admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:04.511085 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0eecc02-ab70-4d87-be08-c6a91d90707f admin admin] RBAC: Authorizing `identity:create_grant(project_id=d8abf6d39eac40c28c62cd3a8a7d250f, user_id=3b2a5747ed0443e1928b637152e4bfdf, role_id=092036ee46ac427baf4d56468575f697)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:04.511277 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b879decc-0c57-43b9-a34f-9c56c7ebc036 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:04.511697 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b879decc-0c57-43b9-a34f-9c56c7ebc036 admin admin] PATH_INFO: `/v3/projects/aad4ea28111941a19ea89bf076cc9100/users/5b2c6b9740a14ee4bfeac6e3c45eec2f/roles/092036ee46ac427baf4d56468575f697` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:04.513760 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b879decc-0c57-43b9-a34f-9c56c7ebc036 admin admin] RBAC: Authorizing `identity:create_grant(project_id=aad4ea28111941a19ea89bf076cc9100, user_id=5b2c6b9740a14ee4bfeac6e3c45eec2f, role_id=092036ee46ac427baf4d56468575f697)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:04.514032 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0eecc02-ab70-4d87-be08-c6a91d90707f admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:04.517669 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b879decc-0c57-43b9-a34f-9c56c7ebc036 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:04.535476 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 992/1955] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:57:04 2022] PUT /identity/v3/projects/d8abf6d39eac40c28c62cd3a8a7d250f/users/3b2a5747ed0443e1928b637152e4bfdf/roles/092036ee46ac427baf4d56468575f697 => generated 0 bytes in 203 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 09 00:57:04.536506 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0eecc02-ab70-4d87-be08-c6a91d90707f admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:04.537281 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 964/1956] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:57:04 2022] PUT /identity/v3/projects/aad4ea28111941a19ea89bf076cc9100/users/5b2c6b9740a14ee4bfeac6e3c45eec2f/roles/092036ee46ac427baf4d56468575f697 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:04.552149 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98c7a5b9-a466-4056-8a96-6c6b96e55aa6 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:04.552864 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98c7a5b9-a466-4056-8a96-6c6b96e55aa6 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:04.553344 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98c7a5b9-a466-4056-8a96-6c6b96e55aa6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:04.613896 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-98c7a5b9-a466-4056-8a96-6c6b96e55aa6 None None] MFA Rules not processed for user `5b2c6b9740a14ee4bfeac6e3c45eec2f`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:04.615112 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-98c7a5b9-a466-4056-8a96-6c6b96e55aa6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:04.618004 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0eecc02-ab70-4d87-be08-c6a91d90707f admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:04.720005 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:04.735455 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 965/1957] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:04 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 09 00:57:04.736580 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf89ea79-4fb6-4627-9fe4-ae7ba61530bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:04.737949 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf89ea79-4fb6-4627-9fe4-ae7ba61530bd admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:04.738239 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf89ea79-4fb6-4627-9fe4-ae7ba61530bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:04.738508 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf89ea79-4fb6-4627-9fe4-ae7ba61530bd admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:04.739828 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf89ea79-4fb6-4627-9fe4-ae7ba61530bd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:04.741872 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf89ea79-4fb6-4627-9fe4-ae7ba61530bd admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:04.757990 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 993/1958] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:57:04 2022] GET /identity/v3/roles => generated 1420 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 09 00:57:04.759085 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98c7a5b9-a466-4056-8a96-6c6b96e55aa6 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:04.770144 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98c7a5b9-a466-4056-8a96-6c6b96e55aa6 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:04.779238 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a9e8710-72e0-4fd4-8dc0-95df080494fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:04.779965 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a9e8710-72e0-4fd4-8dc0-95df080494fb admin admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:04.780249 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a9e8710-72e0-4fd4-8dc0-95df080494fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:04.780546 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a9e8710-72e0-4fd4-8dc0-95df080494fb admin admin] PATH_INFO: `/v3/projects/d8abf6d39eac40c28c62cd3a8a7d250f/users/3b2a5747ed0443e1928b637152e4bfdf/roles/092036ee46ac427baf4d56468575f697` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:04.782712 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a9e8710-72e0-4fd4-8dc0-95df080494fb admin admin] RBAC: Authorizing `identity:create_grant(project_id=d8abf6d39eac40c28c62cd3a8a7d250f, user_id=3b2a5747ed0443e1928b637152e4bfdf, role_id=092036ee46ac427baf4d56468575f697)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:04.786595 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a9e8710-72e0-4fd4-8dc0-95df080494fb admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:04.806803 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 966/1959] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:57:04 2022] PUT /identity/v3/projects/d8abf6d39eac40c28c62cd3a8a7d250f/users/3b2a5747ed0443e1928b637152e4bfdf/roles/092036ee46ac427baf4d56468575f697 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 09 00:57:04.817571 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4b088ef-fd91-4c35-8397-79a0ffc811aa None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:04.817935 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4b088ef-fd91-4c35-8397-79a0ffc811aa None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:04.818506 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4b088ef-fd91-4c35-8397-79a0ffc811aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:04.825106 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a9e8710-72e0-4fd4-8dc0-95df080494fb admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:04.827029 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-6a9e8710-72e0-4fd4-8dc0-95df080494fb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:04.853416 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-b4b088ef-fd91-4c35-8397-79a0ffc811aa None None] MFA Rules not processed for user `3b2a5747ed0443e1928b637152e4bfdf`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:04.854952 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-b4b088ef-fd91-4c35-8397-79a0ffc811aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:04.865214 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:04.901016 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a9e8710-72e0-4fd4-8dc0-95df080494fb admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:04.905449 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-542e2eaa-7bbf-4bec-8e5c-b8ac0f9d804c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3f48335668a4e8f934ac11ff66cd308', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:04.906402 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-542e2eaa-7bbf-4bec-8e5c-b8ac0f9d804c service cinder] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:04.906683 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-542e2eaa-7bbf-4bec-8e5c-b8ac0f9d804c service cinder] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:04.906954 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-542e2eaa-7bbf-4bec-8e5c-b8ac0f9d804c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:04.934214 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 994/1960] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:04 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:57:04.977828 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-542e2eaa-7bbf-4bec-8e5c-b8ac0f9d804c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:04.979186 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-542e2eaa-7bbf-4bec-8e5c-b8ac0f9d804c service cinder] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:04.989276 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 967/1961] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:57:04 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 164 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 09 00:57:05.081348 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4b088ef-fd91-4c35-8397-79a0ffc811aa None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:05.092682 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4b088ef-fd91-4c35-8397-79a0ffc811aa None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:05.098343 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c68aa48f-8ad1-4649-97d7-aa534b18b698 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3f48335668a4e8f934ac11ff66cd308', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:05.098960 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c68aa48f-8ad1-4649-97d7-aa534b18b698 service cinder] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:05.099248 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c68aa48f-8ad1-4649-97d7-aa534b18b698 service cinder] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:05.099510 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c68aa48f-8ad1-4649-97d7-aa534b18b698 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:05.109229 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c68aa48f-8ad1-4649-97d7-aa534b18b698 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:05.110669 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c68aa48f-8ad1-4649-97d7-aa534b18b698 service cinder] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:05.124029 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 995/1962] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:57:05 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:57:05.588622 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-542e2eaa-7bbf-4bec-8e5c-b8ac0f9d804c service cinder] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:05.598955 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-542e2eaa-7bbf-4bec-8e5c-b8ac0f9d804c service cinder] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:05.603334 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c483df8-25ce-45e7-9e1e-e7ca7f875a5a tempest-VolumeQuotaClassesTest-816887380 tempest-VolumeQuotaClassesTest-816887380-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3b2a5747ed0443e1928b637152e4bfdf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd8abf6d39eac40c28c62cd3a8a7d250f', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:05.603956 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c483df8-25ce-45e7-9e1e-e7ca7f875a5a tempest-VolumeQuotaClassesTest-816887380 tempest-VolumeQuotaClassesTest-816887380-project-admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:05.604246 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c483df8-25ce-45e7-9e1e-e7ca7f875a5a tempest-VolumeQuotaClassesTest-816887380 tempest-VolumeQuotaClassesTest-816887380-project-admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:05.604522 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c483df8-25ce-45e7-9e1e-e7ca7f875a5a tempest-VolumeQuotaClassesTest-816887380 tempest-VolumeQuotaClassesTest-816887380-project-admin] PATH_INFO: `/v3/domains` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:05.605187 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c483df8-25ce-45e7-9e1e-e7ca7f875a5a tempest-VolumeQuotaClassesTest-816887380 tempest-VolumeQuotaClassesTest-816887380-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:57:05.605488 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c483df8-25ce-45e7-9e1e-e7ca7f875a5a tempest-VolumeQuotaClassesTest-816887380 tempest-VolumeQuotaClassesTest-816887380-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:05.607058 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c483df8-25ce-45e7-9e1e-e7ca7f875a5a tempest-VolumeQuotaClassesTest-816887380 tempest-VolumeQuotaClassesTest-816887380-project-admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:05.618255 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 968/1963] 10.222.0.33 () {58 vars in 1153 bytes} [Sun Oct 9 00:57:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:57:05.622376 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c68aa48f-8ad1-4649-97d7-aa534b18b698 service cinder] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:05.631872 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c68aa48f-8ad1-4649-97d7-aa534b18b698 service cinder] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:05.636571 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee310469-e7f4-4669-9a9f-2ef3a9613433 tempest-VolumeQuotaClassesTest-816887380 tempest-VolumeQuotaClassesTest-816887380-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3b2a5747ed0443e1928b637152e4bfdf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd8abf6d39eac40c28c62cd3a8a7d250f', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:05.637154 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee310469-e7f4-4669-9a9f-2ef3a9613433 tempest-VolumeQuotaClassesTest-816887380 tempest-VolumeQuotaClassesTest-816887380-project-admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:05.637436 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee310469-e7f4-4669-9a9f-2ef3a9613433 tempest-VolumeQuotaClassesTest-816887380 tempest-VolumeQuotaClassesTest-816887380-project-admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:05.637715 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee310469-e7f4-4669-9a9f-2ef3a9613433 tempest-VolumeQuotaClassesTest-816887380 tempest-VolumeQuotaClassesTest-816887380-project-admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:05.638349 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee310469-e7f4-4669-9a9f-2ef3a9613433 tempest-VolumeQuotaClassesTest-816887380 tempest-VolumeQuotaClassesTest-816887380-project-admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:05.640330 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee310469-e7f4-4669-9a9f-2ef3a9613433 tempest-VolumeQuotaClassesTest-816887380 tempest-VolumeQuotaClassesTest-816887380-project-admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:05.667174 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 996/1964] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 00:57:05 2022] POST /identity/v3/projects => generated 354 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:57:05.671225 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c483df8-25ce-45e7-9e1e-e7ca7f875a5a tempest-VolumeQuotaClassesTest-816887380 tempest-VolumeQuotaClassesTest-816887380-project-admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:05.680893 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c483df8-25ce-45e7-9e1e-e7ca7f875a5a tempest-VolumeQuotaClassesTest-816887380 tempest-VolumeQuotaClassesTest-816887380-project-admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:05.736427 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b58a21f-e7c8-4017-9791-832dff59dda5 tempest-VolumeQuotaClassesTest-816887380 tempest-VolumeQuotaClassesTest-816887380-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3b2a5747ed0443e1928b637152e4bfdf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd8abf6d39eac40c28c62cd3a8a7d250f', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:05.737625 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b58a21f-e7c8-4017-9791-832dff59dda5 tempest-VolumeQuotaClassesTest-816887380 tempest-VolumeQuotaClassesTest-816887380-project-admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:05.737910 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b58a21f-e7c8-4017-9791-832dff59dda5 tempest-VolumeQuotaClassesTest-816887380 tempest-VolumeQuotaClassesTest-816887380-project-admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:05.738192 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b58a21f-e7c8-4017-9791-832dff59dda5 tempest-VolumeQuotaClassesTest-816887380 tempest-VolumeQuotaClassesTest-816887380-project-admin] PATH_INFO: `/v3/domains` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:05.738832 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b58a21f-e7c8-4017-9791-832dff59dda5 tempest-VolumeQuotaClassesTest-816887380 tempest-VolumeQuotaClassesTest-816887380-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:57:05.739149 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b58a21f-e7c8-4017-9791-832dff59dda5 tempest-VolumeQuotaClassesTest-816887380 tempest-VolumeQuotaClassesTest-816887380-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:05.740749 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b58a21f-e7c8-4017-9791-832dff59dda5 tempest-VolumeQuotaClassesTest-816887380 tempest-VolumeQuotaClassesTest-816887380-project-admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:05.752373 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 969/1965] 10.222.0.33 () {58 vars in 1153 bytes} [Sun Oct 9 00:57:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:57:05.789741 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee310469-e7f4-4669-9a9f-2ef3a9613433 tempest-VolumeQuotaClassesTest-816887380 tempest-VolumeQuotaClassesTest-816887380-project-admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:05.800759 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee310469-e7f4-4669-9a9f-2ef3a9613433 tempest-VolumeQuotaClassesTest-816887380 tempest-VolumeQuotaClassesTest-816887380-project-admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:05.805731 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ea288e6-169b-4910-8a45-bf9b2667bf7e tempest-VolumeQuotaClassesTest-816887380 tempest-VolumeQuotaClassesTest-816887380-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3b2a5747ed0443e1928b637152e4bfdf', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'd8abf6d39eac40c28c62cd3a8a7d250f', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:05.806345 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ea288e6-169b-4910-8a45-bf9b2667bf7e tempest-VolumeQuotaClassesTest-816887380 tempest-VolumeQuotaClassesTest-816887380-project-admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:05.806678 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ea288e6-169b-4910-8a45-bf9b2667bf7e tempest-VolumeQuotaClassesTest-816887380 tempest-VolumeQuotaClassesTest-816887380-project-admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:05.806916 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ea288e6-169b-4910-8a45-bf9b2667bf7e tempest-VolumeQuotaClassesTest-816887380 tempest-VolumeQuotaClassesTest-816887380-project-admin] PATH_INFO: `/v3/projects/14d077fe79854a65b09ffe9fb3975d64` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:05.808147 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ea288e6-169b-4910-8a45-bf9b2667bf7e tempest-VolumeQuotaClassesTest-816887380 tempest-VolumeQuotaClassesTest-816887380-project-admin] RBAC: Authorizing `identity:delete_project(project_id=14d077fe79854a65b09ffe9fb3975d64)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:05.810123 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ea288e6-169b-4910-8a45-bf9b2667bf7e tempest-VolumeQuotaClassesTest-816887380 tempest-VolumeQuotaClassesTest-816887380-project-admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:05.857919 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-7ea288e6-169b-4910-8a45-bf9b2667bf7e tempest-VolumeQuotaClassesTest-816887380 tempest-VolumeQuotaClassesTest-816887380-project-admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '14d077fe79854a65b09ffe9fb3975d64', 'request_id': 'req-7ea288e6-169b-4910-8a45-bf9b2667bf7e'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:05.864740 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-7ea288e6-169b-4910-8a45-bf9b2667bf7e tempest-VolumeQuotaClassesTest-816887380 tempest-VolumeQuotaClassesTest-816887380-project-admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '14d077fe79854a65b09ffe9fb3975d64', 'request_id': 'req-7ea288e6-169b-4910-8a45-bf9b2667bf7e'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:05.871310 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-7ea288e6-169b-4910-8a45-bf9b2667bf7e tempest-VolumeQuotaClassesTest-816887380 tempest-VolumeQuotaClassesTest-816887380-project-admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '14d077fe79854a65b09ffe9fb3975d64', 'request_id': 'req-7ea288e6-169b-4910-8a45-bf9b2667bf7e'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:05.885894 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-7ea288e6-169b-4910-8a45-bf9b2667bf7e tempest-VolumeQuotaClassesTest-816887380 tempest-VolumeQuotaClassesTest-816887380-project-admin] The token cache is being invalidate because project 14d077fe79854a65b09ffe9fb3975d64 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 00:57:05.886186 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-7ea288e6-169b-4910-8a45-bf9b2667bf7e tempest-VolumeQuotaClassesTest-816887380 tempest-VolumeQuotaClassesTest-816887380-project-admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:05.886856 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-7ea288e6-169b-4910-8a45-bf9b2667bf7e tempest-VolumeQuotaClassesTest-816887380 tempest-VolumeQuotaClassesTest-816887380-project-admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:05.888403 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 997/1966] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 00:57:05 2022] DELETE /identity/v3/projects/14d077fe79854a65b09ffe9fb3975d64 => generated 0 bytes in 99 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:06.097339 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b58a21f-e7c8-4017-9791-832dff59dda5 tempest-VolumeQuotaClassesTest-816887380 tempest-VolumeQuotaClassesTest-816887380-project-admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:06.099217 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-4b58a21f-e7c8-4017-9791-832dff59dda5 tempest-VolumeQuotaClassesTest-816887380 tempest-VolumeQuotaClassesTest-816887380-project-admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:06.164749 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b58a21f-e7c8-4017-9791-832dff59dda5 tempest-VolumeQuotaClassesTest-816887380 tempest-VolumeQuotaClassesTest-816887380-project-admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:06.168619 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:06.230147 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab593584-c959-497b-a561-73e3e93a8609 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:06.231129 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab593584-c959-497b-a561-73e3e93a8609 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:06.231704 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab593584-c959-497b-a561-73e3e93a8609 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:06.232144 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab593584-c959-497b-a561-73e3e93a8609 admin admin] PATH_INFO: `/v3/users/b536979b218641bab926bb51c84789e0` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:06.233558 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab593584-c959-497b-a561-73e3e93a8609 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b536979b218641bab926bb51c84789e0)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:06.235560 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab593584-c959-497b-a561-73e3e93a8609 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:06.296776 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-ab593584-c959-497b-a561-73e3e93a8609 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b536979b218641bab926bb51c84789e0'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:06.306276 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-ab593584-c959-497b-a561-73e3e93a8609 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b536979b218641bab926bb51c84789e0'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:06.307668 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-ab593584-c959-497b-a561-73e3e93a8609 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b536979b218641bab926bb51c84789e0'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:06.317198 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-ab593584-c959-497b-a561-73e3e93a8609 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b536979b218641bab926bb51c84789e0'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:06.332224 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-ab593584-c959-497b-a561-73e3e93a8609 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b536979b218641bab926bb51c84789e0'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:06.349864 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 970/1967] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 00:57:06 2022] DELETE /identity/v3/users/b536979b218641bab926bb51c84789e0 => generated 0 bytes in 254 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:06.354558 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9db33fb-d134-4e57-9bf6-9a077a752cac None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:06.354962 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9db33fb-d134-4e57-9bf6-9a077a752cac None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:06.355281 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9db33fb-d134-4e57-9bf6-9a077a752cac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:06.377439 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-d9db33fb-d134-4e57-9bf6-9a077a752cac None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:06.378668 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-d9db33fb-d134-4e57-9bf6-9a077a752cac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:06.399930 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:06.450640 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 998/1968] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:57:06.461520 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab593584-c959-497b-a561-73e3e93a8609 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:06.463523 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-ab593584-c959-497b-a561-73e3e93a8609 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:06.531880 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab593584-c959-497b-a561-73e3e93a8609 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:06.536832 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66efcd91-7afb-42f9-871a-28001f8c3f9f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:06.537417 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66efcd91-7afb-42f9-871a-28001f8c3f9f service neutron] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:06.537701 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66efcd91-7afb-42f9-871a-28001f8c3f9f service neutron] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:06.537976 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66efcd91-7afb-42f9-871a-28001f8c3f9f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:06.548479 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66efcd91-7afb-42f9-871a-28001f8c3f9f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:06.550263 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66efcd91-7afb-42f9-871a-28001f8c3f9f service neutron] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:06.564207 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 971/1969] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:57:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 09 00:57:06.598689 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9db33fb-d134-4e57-9bf6-9a077a752cac None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:06.600175 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-d9db33fb-d134-4e57-9bf6-9a077a752cac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:06.665037 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66efcd91-7afb-42f9-871a-28001f8c3f9f service neutron] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:06.666632 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-66efcd91-7afb-42f9-871a-28001f8c3f9f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:06.679957 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9db33fb-d134-4e57-9bf6-9a077a752cac None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:06.682667 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66efcd91-7afb-42f9-871a-28001f8c3f9f service neutron] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:06.688221 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79c70b20-f466-46b8-9853-6dd18f53aaa8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:06.688867 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79c70b20-f466-46b8-9853-6dd18f53aaa8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:06.689580 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79c70b20-f466-46b8-9853-6dd18f53aaa8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:06.689580 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79c70b20-f466-46b8-9853-6dd18f53aaa8 admin admin] PATH_INFO: `/v3/users/63c5754454dc43faa321fb13d863ada9` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:06.690646 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79c70b20-f466-46b8-9853-6dd18f53aaa8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=63c5754454dc43faa321fb13d863ada9)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:06.692355 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e17bcd46-b33b-4ee5-9c23-5c3795fd1d2f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3f48335668a4e8f934ac11ff66cd308', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:06.692772 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79c70b20-f466-46b8-9853-6dd18f53aaa8 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:06.693305 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e17bcd46-b33b-4ee5-9c23-5c3795fd1d2f service cinder] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:06.693745 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e17bcd46-b33b-4ee5-9c23-5c3795fd1d2f service cinder] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:06.694492 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e17bcd46-b33b-4ee5-9c23-5c3795fd1d2f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:06.697065 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-e17bcd46-b33b-4ee5-9c23-5c3795fd1d2f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:06.724336 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:06.753016 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-79c70b20-f466-46b8-9853-6dd18f53aaa8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '63c5754454dc43faa321fb13d863ada9'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:06.764359 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-79c70b20-f466-46b8-9853-6dd18f53aaa8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '63c5754454dc43faa321fb13d863ada9'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:06.765107 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-79c70b20-f466-46b8-9853-6dd18f53aaa8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '63c5754454dc43faa321fb13d863ada9'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:06.774469 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-79c70b20-f466-46b8-9853-6dd18f53aaa8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '63c5754454dc43faa321fb13d863ada9'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:06.792843 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-79c70b20-f466-46b8-9853-6dd18f53aaa8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '63c5754454dc43faa321fb13d863ada9'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:06.801541 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 972/1970] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 00:57:06 2022] DELETE /identity/v3/users/63c5754454dc43faa321fb13d863ada9 => generated 0 bytes in 137 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:06.805478 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-796f7be8-c2ff-483d-a35e-d8bff577b405 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:06.805776 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-796f7be8-c2ff-483d-a35e-d8bff577b405 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:06.806050 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-796f7be8-c2ff-483d-a35e-d8bff577b405 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:06.806819 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e17bcd46-b33b-4ee5-9c23-5c3795fd1d2f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:06.811127 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e17bcd46-b33b-4ee5-9c23-5c3795fd1d2f service cinder] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:06.823445 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 999/1971] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:57:06 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 225 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:57:06.835918 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-796f7be8-c2ff-483d-a35e-d8bff577b405 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:06.837139 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-796f7be8-c2ff-483d-a35e-d8bff577b405 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:06.939814 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 973/1972] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:57:06.947354 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e17bcd46-b33b-4ee5-9c23-5c3795fd1d2f service cinder] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:06.957556 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e17bcd46-b33b-4ee5-9c23-5c3795fd1d2f service cinder] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:07.024701 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37ae96cd-dda8-44ca-a006-e09903980d3d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:07.025987 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37ae96cd-dda8-44ca-a006-e09903980d3d service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:07.026341 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37ae96cd-dda8-44ca-a006-e09903980d3d service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:07.026684 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37ae96cd-dda8-44ca-a006-e09903980d3d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:07.039219 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-796f7be8-c2ff-483d-a35e-d8bff577b405 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:07.041126 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-796f7be8-c2ff-483d-a35e-d8bff577b405 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:07.055955 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37ae96cd-dda8-44ca-a006-e09903980d3d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:07.058580 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37ae96cd-dda8-44ca-a006-e09903980d3d service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:07.062100 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-796f7be8-c2ff-483d-a35e-d8bff577b405 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:07.071816 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb6d62ac-ba66-435e-95ad-a2ee075c0839 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:07.072529 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6d62ac-ba66-435e-95ad-a2ee075c0839 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:07.072811 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6d62ac-ba66-435e-95ad-a2ee075c0839 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:07.073086 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6d62ac-ba66-435e-95ad-a2ee075c0839 admin admin] PATH_INFO: `/v3/projects/c6be54a4bceb4bc8b03ba71439906556` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:07.074709 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb6d62ac-ba66-435e-95ad-a2ee075c0839 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c6be54a4bceb4bc8b03ba71439906556)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:07.076422 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb6d62ac-ba66-435e-95ad-a2ee075c0839 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:07.084770 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1000/1973] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:57:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 138 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:57:07.162846 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-cb6d62ac-ba66-435e-95ad-a2ee075c0839 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c6be54a4bceb4bc8b03ba71439906556', 'request_id': 'req-cb6d62ac-ba66-435e-95ad-a2ee075c0839'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:07.170972 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-cb6d62ac-ba66-435e-95ad-a2ee075c0839 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c6be54a4bceb4bc8b03ba71439906556', 'request_id': 'req-cb6d62ac-ba66-435e-95ad-a2ee075c0839'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:07.179118 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-cb6d62ac-ba66-435e-95ad-a2ee075c0839 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c6be54a4bceb4bc8b03ba71439906556', 'request_id': 'req-cb6d62ac-ba66-435e-95ad-a2ee075c0839'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:07.197129 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-cb6d62ac-ba66-435e-95ad-a2ee075c0839 admin admin] The token cache is being invalidate because project c6be54a4bceb4bc8b03ba71439906556 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81453) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 00:57:07.197360 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-cb6d62ac-ba66-435e-95ad-a2ee075c0839 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:07.198111 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-cb6d62ac-ba66-435e-95ad-a2ee075c0839 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:07.204850 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 974/1974] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 00:57:07 2022] DELETE /identity/v3/projects/c6be54a4bceb4bc8b03ba71439906556 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 09 00:57:07.209114 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37ae96cd-dda8-44ca-a006-e09903980d3d service neutron] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:07.219900 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37ae96cd-dda8-44ca-a006-e09903980d3d service neutron] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:07.265816 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:07.313472 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f47ec3b-3a78-4b00-a322-d5fa0f9ea929 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:07.314126 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f47ec3b-3a78-4b00-a322-d5fa0f9ea929 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:07.314415 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f47ec3b-3a78-4b00-a322-d5fa0f9ea929 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:07.314781 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f47ec3b-3a78-4b00-a322-d5fa0f9ea929 admin admin] PATH_INFO: `/v3/users/3b2a5747ed0443e1928b637152e4bfdf` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:07.317899 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f47ec3b-3a78-4b00-a322-d5fa0f9ea929 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3b2a5747ed0443e1928b637152e4bfdf)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:07.320493 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f47ec3b-3a78-4b00-a322-d5fa0f9ea929 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:07.412189 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-6f47ec3b-3a78-4b00-a322-d5fa0f9ea929 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3b2a5747ed0443e1928b637152e4bfdf'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:07.423178 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-6f47ec3b-3a78-4b00-a322-d5fa0f9ea929 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3b2a5747ed0443e1928b637152e4bfdf'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:07.424125 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-6f47ec3b-3a78-4b00-a322-d5fa0f9ea929 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3b2a5747ed0443e1928b637152e4bfdf'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:07.425484 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-6f47ec3b-3a78-4b00-a322-d5fa0f9ea929 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3b2a5747ed0443e1928b637152e4bfdf'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:07.435255 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-6f47ec3b-3a78-4b00-a322-d5fa0f9ea929 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3b2a5747ed0443e1928b637152e4bfdf'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:07.480554 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1001/1975] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 00:57:07 2022] DELETE /identity/v3/users/3b2a5747ed0443e1928b637152e4bfdf => generated 0 bytes in 267 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:07.648523 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb6d62ac-ba66-435e-95ad-a2ee075c0839 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:07.650162 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-cb6d62ac-ba66-435e-95ad-a2ee075c0839 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:07.683949 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb6d62ac-ba66-435e-95ad-a2ee075c0839 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:07.713950 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=85,keystone:DELETE=13,keystone:INSERT=2 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:07.780713 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2155d1bd-97a7-4287-959c-a661e9d4ddec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:07.781652 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2155d1bd-97a7-4287-959c-a661e9d4ddec admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:07.781929 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2155d1bd-97a7-4287-959c-a661e9d4ddec admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:07.782197 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2155d1bd-97a7-4287-959c-a661e9d4ddec admin admin] PATH_INFO: `/v3/users/1fd3830f6a054cbabb3ca141f2a684f7` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:07.784892 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2155d1bd-97a7-4287-959c-a661e9d4ddec admin admin] RBAC: Authorizing `identity:delete_user(user_id=1fd3830f6a054cbabb3ca141f2a684f7)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:07.786863 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2155d1bd-97a7-4287-959c-a661e9d4ddec admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:07.813797 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f47ec3b-3a78-4b00-a322-d5fa0f9ea929 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:07.815919 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-6f47ec3b-3a78-4b00-a322-d5fa0f9ea929 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:07.829625 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f47ec3b-3a78-4b00-a322-d5fa0f9ea929 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:07.836127 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-919427a4-e3a0-4098-a066-a85acd74b06b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:07.837050 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919427a4-e3a0-4098-a066-a85acd74b06b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:07.837561 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919427a4-e3a0-4098-a066-a85acd74b06b admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:07.838117 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919427a4-e3a0-4098-a066-a85acd74b06b admin admin] PATH_INFO: `/v3/projects/6824f2b206d44e33b37c484dcea0e96f` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:07.839758 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-919427a4-e3a0-4098-a066-a85acd74b06b admin admin] RBAC: Authorizing `identity:delete_project(project_id=6824f2b206d44e33b37c484dcea0e96f)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:07.842846 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-919427a4-e3a0-4098-a066-a85acd74b06b admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:07.877535 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-2155d1bd-97a7-4287-959c-a661e9d4ddec admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1fd3830f6a054cbabb3ca141f2a684f7'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:07.888697 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-2155d1bd-97a7-4287-959c-a661e9d4ddec admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1fd3830f6a054cbabb3ca141f2a684f7'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:07.889839 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-2155d1bd-97a7-4287-959c-a661e9d4ddec admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1fd3830f6a054cbabb3ca141f2a684f7'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:07.901343 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-2155d1bd-97a7-4287-959c-a661e9d4ddec admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1fd3830f6a054cbabb3ca141f2a684f7'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:07.917605 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-2155d1bd-97a7-4287-959c-a661e9d4ddec admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1fd3830f6a054cbabb3ca141f2a684f7'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:07.918826 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-919427a4-e3a0-4098-a066-a85acd74b06b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6824f2b206d44e33b37c484dcea0e96f', 'request_id': 'req-919427a4-e3a0-4098-a066-a85acd74b06b'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:07.928060 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 975/1976] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 00:57:07 2022] DELETE /identity/v3/users/1fd3830f6a054cbabb3ca141f2a684f7 => generated 0 bytes in 277 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:07.931873 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10fef336-2340-4ef4-bdca-9549d249b23a None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:07.932521 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10fef336-2340-4ef4-bdca-9549d249b23a None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:07.932521 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10fef336-2340-4ef4-bdca-9549d249b23a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:07.937321 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-919427a4-e3a0-4098-a066-a85acd74b06b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6824f2b206d44e33b37c484dcea0e96f', 'request_id': 'req-919427a4-e3a0-4098-a066-a85acd74b06b'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:07.945698 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-919427a4-e3a0-4098-a066-a85acd74b06b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6824f2b206d44e33b37c484dcea0e96f', 'request_id': 'req-919427a4-e3a0-4098-a066-a85acd74b06b'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:07.969927 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-10fef336-2340-4ef4-bdca-9549d249b23a None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:07.971138 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-10fef336-2340-4ef4-bdca-9549d249b23a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:07.974150 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-919427a4-e3a0-4098-a066-a85acd74b06b admin admin] The token cache is being invalidate because project 6824f2b206d44e33b37c484dcea0e96f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 00:57:07.974733 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-919427a4-e3a0-4098-a066-a85acd74b06b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:07.975860 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-919427a4-e3a0-4098-a066-a85acd74b06b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:07.979894 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1002/1977] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 00:57:07 2022] DELETE /identity/v3/projects/6824f2b206d44e33b37c484dcea0e96f => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 09 00:57:07.983518 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-919427a4-e3a0-4098-a066-a85acd74b06b admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:07.986681 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-919427a4-e3a0-4098-a066-a85acd74b06b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:08.046903 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 976/1978] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:57:08.066493 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:08.109460 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10fef336-2340-4ef4-bdca-9549d249b23a None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:08.111707 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-10fef336-2340-4ef4-bdca-9549d249b23a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:08.141498 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-919427a4-e3a0-4098-a066-a85acd74b06b admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:08.164407 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c678e3d0-99cd-4c36-a12e-8a970e462bb1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:08.165280 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c678e3d0-99cd-4c36-a12e-8a970e462bb1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:08.165657 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c678e3d0-99cd-4c36-a12e-8a970e462bb1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:08.166306 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c678e3d0-99cd-4c36-a12e-8a970e462bb1 admin admin] PATH_INFO: `/v3/users/facfaba0804b4ce4b3228911bc3747a2` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:08.167824 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c678e3d0-99cd-4c36-a12e-8a970e462bb1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=facfaba0804b4ce4b3228911bc3747a2)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:08.170657 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c678e3d0-99cd-4c36-a12e-8a970e462bb1 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:08.219309 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:08.242601 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10fef336-2340-4ef4-bdca-9549d249b23a None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:08.247910 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9967f0a2-282b-4dee-af50-899a6c7beb27 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:08.249066 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9967f0a2-282b-4dee-af50-899a6c7beb27 service neutron] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:08.250043 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9967f0a2-282b-4dee-af50-899a6c7beb27 service neutron] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:08.258996 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9967f0a2-282b-4dee-af50-899a6c7beb27 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:08.276660 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-c678e3d0-99cd-4c36-a12e-8a970e462bb1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'facfaba0804b4ce4b3228911bc3747a2'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:08.286063 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9967f0a2-282b-4dee-af50-899a6c7beb27 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:08.288659 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9967f0a2-282b-4dee-af50-899a6c7beb27 service neutron] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:08.293645 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-c678e3d0-99cd-4c36-a12e-8a970e462bb1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'facfaba0804b4ce4b3228911bc3747a2'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:08.294472 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-c678e3d0-99cd-4c36-a12e-8a970e462bb1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'facfaba0804b4ce4b3228911bc3747a2'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:08.295702 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-c678e3d0-99cd-4c36-a12e-8a970e462bb1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'facfaba0804b4ce4b3228911bc3747a2'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:08.311931 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 977/1979] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:57:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 203 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:57:08.318372 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-c678e3d0-99cd-4c36-a12e-8a970e462bb1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'facfaba0804b4ce4b3228911bc3747a2'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:08.343466 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1003/1980] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 00:57:07 2022] DELETE /identity/v3/users/facfaba0804b4ce4b3228911bc3747a2 => generated 0 bytes in 358 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 09 00:57:08.934040 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9967f0a2-282b-4dee-af50-899a6c7beb27 service neutron] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:08.935639 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-9967f0a2-282b-4dee-af50-899a6c7beb27 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:08.950466 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9967f0a2-282b-4dee-af50-899a6c7beb27 service neutron] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:09.063810 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f74ce08-8ba0-43fe-ab84-f1ef1d68cb86 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:09.064462 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f74ce08-8ba0-43fe-ab84-f1ef1d68cb86 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:09.064745 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f74ce08-8ba0-43fe-ab84-f1ef1d68cb86 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:09.065035 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f74ce08-8ba0-43fe-ab84-f1ef1d68cb86 admin admin] PATH_INFO: `/v3/projects/d8abf6d39eac40c28c62cd3a8a7d250f` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:09.066871 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f74ce08-8ba0-43fe-ab84-f1ef1d68cb86 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d8abf6d39eac40c28c62cd3a8a7d250f)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:09.069507 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f74ce08-8ba0-43fe-ab84-f1ef1d68cb86 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:09.168129 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c678e3d0-99cd-4c36-a12e-8a970e462bb1 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:09.171377 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-c678e3d0-99cd-4c36-a12e-8a970e462bb1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:09.195484 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c678e3d0-99cd-4c36-a12e-8a970e462bb1 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:09.240630 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-8f74ce08-8ba0-43fe-ab84-f1ef1d68cb86 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd8abf6d39eac40c28c62cd3a8a7d250f', 'request_id': 'req-8f74ce08-8ba0-43fe-ab84-f1ef1d68cb86'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:09.247527 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-8f74ce08-8ba0-43fe-ab84-f1ef1d68cb86 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd8abf6d39eac40c28c62cd3a8a7d250f', 'request_id': 'req-8f74ce08-8ba0-43fe-ab84-f1ef1d68cb86'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:09.257468 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-8f74ce08-8ba0-43fe-ab84-f1ef1d68cb86 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd8abf6d39eac40c28c62cd3a8a7d250f', 'request_id': 'req-8f74ce08-8ba0-43fe-ab84-f1ef1d68cb86'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:09.295440 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-8f74ce08-8ba0-43fe-ab84-f1ef1d68cb86 admin admin] The token cache is being invalidate because project d8abf6d39eac40c28c62cd3a8a7d250f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81453) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 00:57:09.295791 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-8f74ce08-8ba0-43fe-ab84-f1ef1d68cb86 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:09.297427 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-8f74ce08-8ba0-43fe-ab84-f1ef1d68cb86 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:09.299717 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 978/1981] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 00:57:08 2022] DELETE /identity/v3/projects/d8abf6d39eac40c28c62cd3a8a7d250f => generated 0 bytes in 366 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:09.303957 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d32ce4c-c10d-4766-b1f4-62494f87261e None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:09.304491 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d32ce4c-c10d-4766-b1f4-62494f87261e None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:09.304924 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d32ce4c-c10d-4766-b1f4-62494f87261e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:09.322210 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af519a02-b704-4bb0-b8c5-528739a8a36b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:09.323189 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af519a02-b704-4bb0-b8c5-528739a8a36b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:09.323508 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af519a02-b704-4bb0-b8c5-528739a8a36b admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:09.323822 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af519a02-b704-4bb0-b8c5-528739a8a36b admin admin] PATH_INFO: `/v3/projects/349ac70969174f8db0577dc4a138970e` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:09.324969 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af519a02-b704-4bb0-b8c5-528739a8a36b admin admin] RBAC: Authorizing `identity:delete_project(project_id=349ac70969174f8db0577dc4a138970e)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:09.326988 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af519a02-b704-4bb0-b8c5-528739a8a36b admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:09.333874 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-8d32ce4c-c10d-4766-b1f4-62494f87261e None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:09.335732 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-8d32ce4c-c10d-4766-b1f4-62494f87261e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:09.345721 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 979/1982] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:57:09.347032 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d32ce4c-c10d-4766-b1f4-62494f87261e None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:09.349081 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-8d32ce4c-c10d-4766-b1f4-62494f87261e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:09.380439 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d32ce4c-c10d-4766-b1f4-62494f87261e None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:09.386078 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84412c78-7f19-4b68-8dba-d82e782ebe0a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:09.386912 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84412c78-7f19-4b68-8dba-d82e782ebe0a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:09.387199 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84412c78-7f19-4b68-8dba-d82e782ebe0a admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:09.387476 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84412c78-7f19-4b68-8dba-d82e782ebe0a admin admin] PATH_INFO: `/v3/projects/5260f11ec8f645cba5394181e4d191cf` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:09.397479 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84412c78-7f19-4b68-8dba-d82e782ebe0a admin admin] RBAC: Authorizing `identity:delete_project(project_id=5260f11ec8f645cba5394181e4d191cf)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:09.400262 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84412c78-7f19-4b68-8dba-d82e782ebe0a admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:09.424465 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:09.428652 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-af519a02-b704-4bb0-b8c5-528739a8a36b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '349ac70969174f8db0577dc4a138970e', 'request_id': 'req-af519a02-b704-4bb0-b8c5-528739a8a36b'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:09.442353 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-af519a02-b704-4bb0-b8c5-528739a8a36b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '349ac70969174f8db0577dc4a138970e', 'request_id': 'req-af519a02-b704-4bb0-b8c5-528739a8a36b'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:09.453867 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-af519a02-b704-4bb0-b8c5-528739a8a36b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '349ac70969174f8db0577dc4a138970e', 'request_id': 'req-af519a02-b704-4bb0-b8c5-528739a8a36b'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:09.474549 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-af519a02-b704-4bb0-b8c5-528739a8a36b admin admin] The token cache is being invalidate because project 349ac70969174f8db0577dc4a138970e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 00:57:09.474907 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-af519a02-b704-4bb0-b8c5-528739a8a36b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:09.475969 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-af519a02-b704-4bb0-b8c5-528739a8a36b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:09.481990 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1004/1983] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 00:57:09 2022] DELETE /identity/v3/projects/349ac70969174f8db0577dc4a138970e => generated 0 bytes in 311 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:09.483326 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1963d653-57ec-4130-b450-caaf7cec682a None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:09.483638 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1963d653-57ec-4130-b450-caaf7cec682a None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:09.483928 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1963d653-57ec-4130-b450-caaf7cec682a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:09.486194 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-84412c78-7f19-4b68-8dba-d82e782ebe0a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5260f11ec8f645cba5394181e4d191cf', 'request_id': 'req-84412c78-7f19-4b68-8dba-d82e782ebe0a'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:09.492969 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-84412c78-7f19-4b68-8dba-d82e782ebe0a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5260f11ec8f645cba5394181e4d191cf', 'request_id': 'req-84412c78-7f19-4b68-8dba-d82e782ebe0a'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:09.500924 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-84412c78-7f19-4b68-8dba-d82e782ebe0a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5260f11ec8f645cba5394181e4d191cf', 'request_id': 'req-84412c78-7f19-4b68-8dba-d82e782ebe0a'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:09.504186 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=85,keystone:DELETE=13,keystone:INSERT=2 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:09.505282 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-1963d653-57ec-4130-b450-caaf7cec682a None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:09.506742 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-1963d653-57ec-4130-b450-caaf7cec682a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:09.521681 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-84412c78-7f19-4b68-8dba-d82e782ebe0a admin admin] The token cache is being invalidate because project 5260f11ec8f645cba5394181e4d191cf was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81453) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 00:57:09.522253 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-84412c78-7f19-4b68-8dba-d82e782ebe0a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:09.525013 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-84412c78-7f19-4b68-8dba-d82e782ebe0a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:09.527751 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 980/1984] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 00:57:09 2022] DELETE /identity/v3/projects/5260f11ec8f645cba5394181e4d191cf => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:09.529272 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e46228e7-5a10-4b1c-bc43-0e5ef92e83d1 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:09.529738 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e46228e7-5a10-4b1c-bc43-0e5ef92e83d1 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:09.530214 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e46228e7-5a10-4b1c-bc43-0e5ef92e83d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:09.566624 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-e46228e7-5a10-4b1c-bc43-0e5ef92e83d1 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:09.569782 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-e46228e7-5a10-4b1c-bc43-0e5ef92e83d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:09.607635 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1005/1985] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:57:09.609092 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1963d653-57ec-4130-b450-caaf7cec682a None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:09.611925 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-1963d653-57ec-4130-b450-caaf7cec682a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:09.626015 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1963d653-57ec-4130-b450-caaf7cec682a None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:09.632164 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c679df5-869d-4bee-88e6-7f08fd5acd02 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:09.633037 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c679df5-869d-4bee-88e6-7f08fd5acd02 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:09.633772 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c679df5-869d-4bee-88e6-7f08fd5acd02 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:09.634181 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c679df5-869d-4bee-88e6-7f08fd5acd02 admin admin] PATH_INFO: `/v3/users/4fae9d1c91c046378b08c390d58da7d0` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:09.635860 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c679df5-869d-4bee-88e6-7f08fd5acd02 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4fae9d1c91c046378b08c390d58da7d0)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:09.637969 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c679df5-869d-4bee-88e6-7f08fd5acd02 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:09.662599 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 981/1986] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:57:09.662599 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e46228e7-5a10-4b1c-bc43-0e5ef92e83d1 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:09.679550 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e46228e7-5a10-4b1c-bc43-0e5ef92e83d1 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:09.686628 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e11a83ec-314f-4d9e-985a-4dc2e1c18926 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:09.687742 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e11a83ec-314f-4d9e-985a-4dc2e1c18926 admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:09.687859 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e11a83ec-314f-4d9e-985a-4dc2e1c18926 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:09.688236 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e11a83ec-314f-4d9e-985a-4dc2e1c18926 admin admin] PATH_INFO: `/v3/domains` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:09.688971 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e11a83ec-314f-4d9e-985a-4dc2e1c18926 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:57:09.689346 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e11a83ec-314f-4d9e-985a-4dc2e1c18926 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:09.691693 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e11a83ec-314f-4d9e-985a-4dc2e1c18926 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:09.711492 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 982/1987] 10.222.0.33 () {58 vars in 1153 bytes} [Sun Oct 9 00:57:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:57:09.715124 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c48554-56fe-4f34-99a7-18aad7ff0b2d None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:09.715495 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c48554-56fe-4f34-99a7-18aad7ff0b2d None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:09.715865 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c48554-56fe-4f34-99a7-18aad7ff0b2d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:09.738240 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-6c679df5-869d-4bee-88e6-7f08fd5acd02 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4fae9d1c91c046378b08c390d58da7d0'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:09.742767 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-92c48554-56fe-4f34-99a7-18aad7ff0b2d None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:09.744110 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-92c48554-56fe-4f34-99a7-18aad7ff0b2d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:09.753744 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-6c679df5-869d-4bee-88e6-7f08fd5acd02 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4fae9d1c91c046378b08c390d58da7d0'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:09.757781 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 983/1988] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:57:09.757781 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92c48554-56fe-4f34-99a7-18aad7ff0b2d None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:09.761090 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-6c679df5-869d-4bee-88e6-7f08fd5acd02 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4fae9d1c91c046378b08c390d58da7d0'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:09.763215 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-6c679df5-869d-4bee-88e6-7f08fd5acd02 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4fae9d1c91c046378b08c390d58da7d0'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:09.779315 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-6c679df5-869d-4bee-88e6-7f08fd5acd02 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4fae9d1c91c046378b08c390d58da7d0'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:09.789870 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92c48554-56fe-4f34-99a7-18aad7ff0b2d None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:09.801013 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0f9ba85-cb15-465c-9b5b-edc7c448aeb9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:09.801759 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f9ba85-cb15-465c-9b5b-edc7c448aeb9 admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:09.802264 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f9ba85-cb15-465c-9b5b-edc7c448aeb9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:09.802755 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f9ba85-cb15-465c-9b5b-edc7c448aeb9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:09.803655 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0f9ba85-cb15-465c-9b5b-edc7c448aeb9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:09.806150 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0f9ba85-cb15-465c-9b5b-edc7c448aeb9 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:09.825948 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1006/1989] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 00:57:09 2022] DELETE /identity/v3/users/4fae9d1c91c046378b08c390d58da7d0 => generated 0 bytes in 215 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:09.825948 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c679df5-869d-4bee-88e6-7f08fd5acd02 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:09.825948 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-6c679df5-869d-4bee-88e6-7f08fd5acd02 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:09.873036 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 984/1990] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 00:57:09 2022] POST /identity/v3/projects => generated 387 bytes in 118 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:57:09.877001 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0f9ba85-cb15-465c-9b5b-edc7c448aeb9 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:09.885245 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-f0f9ba85-cb15-465c-9b5b-edc7c448aeb9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:09.919616 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:DELETE=14,keystone:SELECT=84,keystone:INSERT=2 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:09.948835 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c679df5-869d-4bee-88e6-7f08fd5acd02 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:10.026205 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebc4c6d7-a0fa-4b3a-bae6-48684a00213a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:10.027152 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc4c6d7-a0fa-4b3a-bae6-48684a00213a admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:10.027560 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc4c6d7-a0fa-4b3a-bae6-48684a00213a admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:10.028317 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc4c6d7-a0fa-4b3a-bae6-48684a00213a admin admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:10.029481 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebc4c6d7-a0fa-4b3a-bae6-48684a00213a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:57:10.029890 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebc4c6d7-a0fa-4b3a-bae6-48684a00213a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:10.032529 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebc4c6d7-a0fa-4b3a-bae6-48684a00213a admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:10.060019 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1007/1991] 10.222.0.33 () {58 vars in 1153 bytes} [Sun Oct 9 00:57:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:57:10.060896 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebc4c6d7-a0fa-4b3a-bae6-48684a00213a admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:10.091171 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebc4c6d7-a0fa-4b3a-bae6-48684a00213a admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:10.099406 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62eafe47-9191-408a-8a12-e96d09cdfee9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:10.100260 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62eafe47-9191-408a-8a12-e96d09cdfee9 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:10.100538 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62eafe47-9191-408a-8a12-e96d09cdfee9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:10.100807 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62eafe47-9191-408a-8a12-e96d09cdfee9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:10.101446 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62eafe47-9191-408a-8a12-e96d09cdfee9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:10.104623 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62eafe47-9191-408a-8a12-e96d09cdfee9 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:10.126955 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0f9ba85-cb15-465c-9b5b-edc7c448aeb9 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:10.142531 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-981073cf-90c7-4e08-938e-b54ba2a7a3cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:10.143699 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-981073cf-90c7-4e08-938e-b54ba2a7a3cc admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:10.144142 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-981073cf-90c7-4e08-938e-b54ba2a7a3cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:10.144652 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-981073cf-90c7-4e08-938e-b54ba2a7a3cc admin admin] PATH_INFO: `/v3/users` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:10.146139 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-981073cf-90c7-4e08-938e-b54ba2a7a3cc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:10.149684 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-981073cf-90c7-4e08-938e-b54ba2a7a3cc admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:10.157309 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1008/1992] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 00:57:10 2022] POST /identity/v3/projects => generated 387 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:57:10.162373 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62eafe47-9191-408a-8a12-e96d09cdfee9 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:10.195207 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 985/1993] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:57:09 2022] POST /identity/v3/users => generated 294 bytes in 318 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:57:10.199224 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-981073cf-90c7-4e08-938e-b54ba2a7a3cc admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:10.222101 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-981073cf-90c7-4e08-938e-b54ba2a7a3cc admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:10.278253 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62eafe47-9191-408a-8a12-e96d09cdfee9 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:10.286656 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddd4f2e7-6dcd-418a-b6c2-5c79d2b32530 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:10.287627 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd4f2e7-6dcd-418a-b6c2-5c79d2b32530 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:10.288583 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd4f2e7-6dcd-418a-b6c2-5c79d2b32530 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:10.289024 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd4f2e7-6dcd-418a-b6c2-5c79d2b32530 admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:10.290683 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddd4f2e7-6dcd-418a-b6c2-5c79d2b32530 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:10.293262 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:DELETE=1,keystone:SELECT=95,keystone:INSERT=4 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:10.294380 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddd4f2e7-6dcd-418a-b6c2-5c79d2b32530 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:10.344668 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e348e0b-bfcb-479b-bed8-51cb9b2adefc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:10.345552 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e348e0b-bfcb-479b-bed8-51cb9b2adefc admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:10.345959 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e348e0b-bfcb-479b-bed8-51cb9b2adefc admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:10.346368 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e348e0b-bfcb-479b-bed8-51cb9b2adefc admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:10.347805 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e348e0b-bfcb-479b-bed8-51cb9b2adefc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:10.348111 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1009/1994] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:57:10 2022] POST /identity/v3/users => generated 294 bytes in 186 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 09 00:57:10.349628 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddd4f2e7-6dcd-418a-b6c2-5c79d2b32530 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:10.350348 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e348e0b-bfcb-479b-bed8-51cb9b2adefc admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:10.365225 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddd4f2e7-6dcd-418a-b6c2-5c79d2b32530 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:10.369939 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 986/1995] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:57:10 2022] GET /identity/v3/roles => generated 1420 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 09 00:57:10.369939 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e348e0b-bfcb-479b-bed8-51cb9b2adefc admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:10.370890 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8c3e051-e175-4756-823c-c12bb1e0e561 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:10.371514 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8c3e051-e175-4756-823c-c12bb1e0e561 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:10.371778 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8c3e051-e175-4756-823c-c12bb1e0e561 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:10.372060 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8c3e051-e175-4756-823c-c12bb1e0e561 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:10.372689 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8c3e051-e175-4756-823c-c12bb1e0e561 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:10.373113 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-8e348e0b-bfcb-479b-bed8-51cb9b2adefc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:10.374292 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8c3e051-e175-4756-823c-c12bb1e0e561 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:10.394115 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1010/1996] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:57:10 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 09 00:57:10.395304 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8c3e051-e175-4756-823c-c12bb1e0e561 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:10.405610 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e348e0b-bfcb-479b-bed8-51cb9b2adefc admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:10.413642 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4b6423c-8103-4a50-9fce-aaf8c4367c13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:10.414303 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4b6423c-8103-4a50-9fce-aaf8c4367c13 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:10.414636 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4b6423c-8103-4a50-9fce-aaf8c4367c13 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:10.415890 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4b6423c-8103-4a50-9fce-aaf8c4367c13 admin admin] PATH_INFO: `/v3/projects/69bf73bbee1444febc675e813fd9af0c` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:10.417090 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4b6423c-8103-4a50-9fce-aaf8c4367c13 admin admin] RBAC: Authorizing `identity:delete_project(project_id=69bf73bbee1444febc675e813fd9af0c)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:10.418616 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8c3e051-e175-4756-823c-c12bb1e0e561 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:10.419009 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4b6423c-8103-4a50-9fce-aaf8c4367c13 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:10.425323 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eb3a7d9-8679-4eb0-bda1-6f4d532ed2ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:10.426195 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eb3a7d9-8679-4eb0-bda1-6f4d532ed2ae admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:10.426651 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eb3a7d9-8679-4eb0-bda1-6f4d532ed2ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:10.427099 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eb3a7d9-8679-4eb0-bda1-6f4d532ed2ae admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:10.427951 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eb3a7d9-8679-4eb0-bda1-6f4d532ed2ae admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:10.429725 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eb3a7d9-8679-4eb0-bda1-6f4d532ed2ae admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:10.449100 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1011/1997] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:57:10 2022] GET /identity/v3/roles => generated 1420 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:57:10.454509 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eb3a7d9-8679-4eb0-bda1-6f4d532ed2ae admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:10.470048 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eb3a7d9-8679-4eb0-bda1-6f4d532ed2ae admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:10.476935 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d014d146-a021-4788-8ad1-367bf293d294 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:10.478233 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d014d146-a021-4788-8ad1-367bf293d294 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:10.478233 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d014d146-a021-4788-8ad1-367bf293d294 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:10.478672 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d014d146-a021-4788-8ad1-367bf293d294 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:10.480016 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d014d146-a021-4788-8ad1-367bf293d294 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:10.481620 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d014d146-a021-4788-8ad1-367bf293d294 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:10.503937 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1012/1998] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:57:10 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:57:10.505059 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d014d146-a021-4788-8ad1-367bf293d294 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:10.512954 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-e4b6423c-8103-4a50-9fce-aaf8c4367c13 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '69bf73bbee1444febc675e813fd9af0c', 'request_id': 'req-e4b6423c-8103-4a50-9fce-aaf8c4367c13'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:10.516585 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d014d146-a021-4788-8ad1-367bf293d294 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:10.520994 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-e4b6423c-8103-4a50-9fce-aaf8c4367c13 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '69bf73bbee1444febc675e813fd9af0c', 'request_id': 'req-e4b6423c-8103-4a50-9fce-aaf8c4367c13'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:10.530897 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-e4b6423c-8103-4a50-9fce-aaf8c4367c13 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '69bf73bbee1444febc675e813fd9af0c', 'request_id': 'req-e4b6423c-8103-4a50-9fce-aaf8c4367c13'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:10.551269 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:10.555633 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-e4b6423c-8103-4a50-9fce-aaf8c4367c13 admin admin] The token cache is being invalidate because project 69bf73bbee1444febc675e813fd9af0c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81453) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 00:57:10.556131 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-e4b6423c-8103-4a50-9fce-aaf8c4367c13 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:10.557149 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-e4b6423c-8103-4a50-9fce-aaf8c4367c13 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:10.559408 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 987/1999] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 00:57:10 2022] DELETE /identity/v3/projects/69bf73bbee1444febc675e813fd9af0c => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:10.560771 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4b6423c-8103-4a50-9fce-aaf8c4367c13 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:10.562560 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-e4b6423c-8103-4a50-9fce-aaf8c4367c13 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:10.605933 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6461e17-5af5-4b12-ac4f-dc9f17c9e2f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:10.606855 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6461e17-5af5-4b12-ac4f-dc9f17c9e2f2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:10.607004 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6461e17-5af5-4b12-ac4f-dc9f17c9e2f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:10.607525 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6461e17-5af5-4b12-ac4f-dc9f17c9e2f2 admin admin] PATH_INFO: `/v3/projects/88ea5df9d105498d896615626269e56a/users/785b47656c7747328d3e6235f86968a4/roles/588016933bab436d8b7c3ac76827356d` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:10.626350 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6461e17-5af5-4b12-ac4f-dc9f17c9e2f2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=88ea5df9d105498d896615626269e56a, user_id=785b47656c7747328d3e6235f86968a4, role_id=588016933bab436d8b7c3ac76827356d)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:10.629814 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6461e17-5af5-4b12-ac4f-dc9f17c9e2f2 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:10.646265 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1013/2000] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:57:10 2022] PUT /identity/v3/projects/88ea5df9d105498d896615626269e56a/users/785b47656c7747328d3e6235f86968a4/roles/588016933bab436d8b7c3ac76827356d => generated 0 bytes in 142 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:10.647640 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0b71e6-0e2a-4fc0-99c6-cc859bb82669 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:10.648002 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0b71e6-0e2a-4fc0-99c6-cc859bb82669 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:10.648186 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0b71e6-0e2a-4fc0-99c6-cc859bb82669 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:10.663893 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4b6423c-8103-4a50-9fce-aaf8c4367c13 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:10.668867 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-3f0b71e6-0e2a-4fc0-99c6-cc859bb82669 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:10.670108 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-3f0b71e6-0e2a-4fc0-99c6-cc859bb82669 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:10.729841 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6484264-1e17-4bf8-aa30-758ed2b3c42a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:10.730515 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6484264-1e17-4bf8-aa30-758ed2b3c42a admin admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:10.730804 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6484264-1e17-4bf8-aa30-758ed2b3c42a admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:10.731089 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6484264-1e17-4bf8-aa30-758ed2b3c42a admin admin] PATH_INFO: `/v3/projects/fb856bc06ad849b1b275840b69eb18a4/users/9a897077384a4ee8a1c87d12dfb8148c/roles/588016933bab436d8b7c3ac76827356d` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:10.745839 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:10.758999 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6484264-1e17-4bf8-aa30-758ed2b3c42a admin admin] RBAC: Authorizing `identity:create_grant(project_id=fb856bc06ad849b1b275840b69eb18a4, user_id=9a897077384a4ee8a1c87d12dfb8148c, role_id=588016933bab436d8b7c3ac76827356d)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:10.763764 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6484264-1e17-4bf8-aa30-758ed2b3c42a admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:10.780485 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1014/2001] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:57:10.780485 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2abe61c1-3311-443d-b6e4-71e63f1f2c4d None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:10.780485 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2abe61c1-3311-443d-b6e4-71e63f1f2c4d None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:10.780485 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2abe61c1-3311-443d-b6e4-71e63f1f2c4d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:10.805438 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 988/2002] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:57:10 2022] PUT /identity/v3/projects/fb856bc06ad849b1b275840b69eb18a4/users/9a897077384a4ee8a1c87d12dfb8148c/roles/588016933bab436d8b7c3ac76827356d => generated 0 bytes in 245 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:10.818045 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-584a7be5-482c-48f9-829a-66d345e70666 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:10.818481 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-584a7be5-482c-48f9-829a-66d345e70666 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:10.818948 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-584a7be5-482c-48f9-829a-66d345e70666 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:10.848437 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-2abe61c1-3311-443d-b6e4-71e63f1f2c4d None None] MFA Rules not processed for user `785b47656c7747328d3e6235f86968a4`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:10.850936 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-2abe61c1-3311-443d-b6e4-71e63f1f2c4d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:10.851060 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-584a7be5-482c-48f9-829a-66d345e70666 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:10.851654 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-584a7be5-482c-48f9-829a-66d345e70666 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:10.934279 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 989/2003] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:57:10.935904 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79f3e8ca-7a97-474d-b55f-cfa9f7f740da None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:10.936343 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79f3e8ca-7a97-474d-b55f-cfa9f7f740da None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:10.936535 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1015/2004] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:10 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:57:10.936743 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79f3e8ca-7a97-474d-b55f-cfa9f7f740da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:10.938100 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2abe61c1-3311-443d-b6e4-71e63f1f2c4d None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:10.952502 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2abe61c1-3311-443d-b6e4-71e63f1f2c4d None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:10.970570 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb3aeddf-12ca-47db-9e1a-ae884e926074 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:10.971025 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-79f3e8ca-7a97-474d-b55f-cfa9f7f740da None None] MFA Rules not processed for user `9a897077384a4ee8a1c87d12dfb8148c`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:10.971841 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3aeddf-12ca-47db-9e1a-ae884e926074 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:10.972130 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3aeddf-12ca-47db-9e1a-ae884e926074 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:10.972380 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-79f3e8ca-7a97-474d-b55f-cfa9f7f740da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:10.972492 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3aeddf-12ca-47db-9e1a-ae884e926074 admin admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:10.973062 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb3aeddf-12ca-47db-9e1a-ae884e926074 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:57:10.973368 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb3aeddf-12ca-47db-9e1a-ae884e926074 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:10.975176 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb3aeddf-12ca-47db-9e1a-ae884e926074 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:10.988087 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1016/2005] 10.222.0.33 () {58 vars in 1153 bytes} [Sun Oct 9 00:57:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:57:10.988736 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb3aeddf-12ca-47db-9e1a-ae884e926074 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:10.990377 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-cb3aeddf-12ca-47db-9e1a-ae884e926074 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:11.004436 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb3aeddf-12ca-47db-9e1a-ae884e926074 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:11.010293 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d1af3f-0198-4be8-9348-d04db00d18bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:11.010904 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d1af3f-0198-4be8-9348-d04db00d18bb admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:11.011184 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d1af3f-0198-4be8-9348-d04db00d18bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:11.011455 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d1af3f-0198-4be8-9348-d04db00d18bb admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:11.012116 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04d1af3f-0198-4be8-9348-d04db00d18bb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:11.014455 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04d1af3f-0198-4be8-9348-d04db00d18bb admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:11.038785 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:11.049139 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 990/2006] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:10 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:57:11.049139 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79f3e8ca-7a97-474d-b55f-cfa9f7f740da None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:11.049419 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1017/2007] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 00:57:10 2022] POST /identity/v3/projects => generated 389 bytes in 58 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:57:11.050729 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d1af3f-0198-4be8-9348-d04db00d18bb admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:11.055559 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79f3e8ca-7a97-474d-b55f-cfa9f7f740da None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:11.139485 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d1af3f-0198-4be8-9348-d04db00d18bb admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:11.145286 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8181c98d-d609-4cc7-a8d7-344d38fb00f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:11.145971 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8181c98d-d609-4cc7-a8d7-344d38fb00f0 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:11.146480 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8181c98d-d609-4cc7-a8d7-344d38fb00f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:11.146834 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8181c98d-d609-4cc7-a8d7-344d38fb00f0 admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:11.147550 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8181c98d-d609-4cc7-a8d7-344d38fb00f0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:11.149562 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8181c98d-d609-4cc7-a8d7-344d38fb00f0 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:11.152926 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7516e66f-68cf-450b-bf3e-b4c30a3dfcc5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:11.153546 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7516e66f-68cf-450b-bf3e-b4c30a3dfcc5 admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:11.154214 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7516e66f-68cf-450b-bf3e-b4c30a3dfcc5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:11.154526 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7516e66f-68cf-450b-bf3e-b4c30a3dfcc5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:11.155164 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7516e66f-68cf-450b-bf3e-b4c30a3dfcc5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:11.157410 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7516e66f-68cf-450b-bf3e-b4c30a3dfcc5 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:11.179678 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1018/2008] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:57:11 2022] POST /identity/v3/users => generated 301 bytes in 130 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:57:11.180958 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8181c98d-d609-4cc7-a8d7-344d38fb00f0 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:11.193853 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8181c98d-d609-4cc7-a8d7-344d38fb00f0 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:11.199065 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 991/2009] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 00:57:11 2022] POST /identity/v3/projects => generated 379 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:57:11.199065 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7516e66f-68cf-450b-bf3e-b4c30a3dfcc5 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:11.208889 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:11.266851 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-361b53d5-b1ac-4c87-8588-54ca7cb51188 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:11.267406 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7516e66f-68cf-450b-bf3e-b4c30a3dfcc5 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:11.268848 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361b53d5-b1ac-4c87-8588-54ca7cb51188 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:11.269139 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361b53d5-b1ac-4c87-8588-54ca7cb51188 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:11.269412 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361b53d5-b1ac-4c87-8588-54ca7cb51188 admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:11.270049 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-361b53d5-b1ac-4c87-8588-54ca7cb51188 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:11.272328 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43ece350-9cb7-4274-94fd-bfed73baf272 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:11.272691 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-361b53d5-b1ac-4c87-8588-54ca7cb51188 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:11.272954 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ece350-9cb7-4274-94fd-bfed73baf272 admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:11.273242 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ece350-9cb7-4274-94fd-bfed73baf272 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:11.273516 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ece350-9cb7-4274-94fd-bfed73baf272 admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:11.274156 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43ece350-9cb7-4274-94fd-bfed73baf272 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:11.275771 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43ece350-9cb7-4274-94fd-bfed73baf272 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:11.289811 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 992/2010] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:57:11 2022] GET /identity/v3/roles => generated 1420 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:57:11.289811 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43ece350-9cb7-4274-94fd-bfed73baf272 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:11.299498 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43ece350-9cb7-4274-94fd-bfed73baf272 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:11.305354 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa7609ea-b23c-400b-b6f3-2077e3b29118 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:11.305898 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa7609ea-b23c-400b-b6f3-2077e3b29118 admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:11.307826 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa7609ea-b23c-400b-b6f3-2077e3b29118 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:11.307826 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa7609ea-b23c-400b-b6f3-2077e3b29118 admin admin] PATH_INFO: `/v3/users` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:11.307826 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa7609ea-b23c-400b-b6f3-2077e3b29118 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:11.309033 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa7609ea-b23c-400b-b6f3-2077e3b29118 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:11.323988 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1019/2011] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 00:57:11 2022] POST /identity/v3/projects => generated 387 bytes in 144 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:57:11.324833 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-361b53d5-b1ac-4c87-8588-54ca7cb51188 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:11.334368 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 993/2012] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:57:11 2022] POST /identity/v3/users => generated 290 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:57:11.338187 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa7609ea-b23c-400b-b6f3-2077e3b29118 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:11.339891 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-aa7609ea-b23c-400b-b6f3-2077e3b29118 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:11.387758 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-361b53d5-b1ac-4c87-8588-54ca7cb51188 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:11.392351 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fb98c47-5a6e-40cd-ac7f-3b82fa61dd72 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:11.393056 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fb98c47-5a6e-40cd-ac7f-3b82fa61dd72 admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:11.393353 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fb98c47-5a6e-40cd-ac7f-3b82fa61dd72 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:11.393626 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fb98c47-5a6e-40cd-ac7f-3b82fa61dd72 admin admin] PATH_INFO: `/v3/projects/9f95152a78a1496ca4266c3255a3e533/users/146df20a8829481f8e6f35708e3debb8/roles/092036ee46ac427baf4d56468575f697` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:11.397609 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa7609ea-b23c-400b-b6f3-2077e3b29118 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:11.407796 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fb98c47-5a6e-40cd-ac7f-3b82fa61dd72 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9f95152a78a1496ca4266c3255a3e533, user_id=146df20a8829481f8e6f35708e3debb8, role_id=092036ee46ac427baf4d56468575f697)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:11.410953 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fb98c47-5a6e-40cd-ac7f-3b82fa61dd72 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:11.419093 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:11.427153 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1020/2013] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:57:11 2022] PUT /identity/v3/projects/9f95152a78a1496ca4266c3255a3e533/users/146df20a8829481f8e6f35708e3debb8/roles/092036ee46ac427baf4d56468575f697 => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:11.430838 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fb98c47-5a6e-40cd-ac7f-3b82fa61dd72 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:11.448054 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8710991-8fcc-4789-a440-2212920996b8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3f48335668a4e8f934ac11ff66cd308', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:11.448669 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8710991-8fcc-4789-a440-2212920996b8 service cinder] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:11.448946 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8710991-8fcc-4789-a440-2212920996b8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:11.449216 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8710991-8fcc-4789-a440-2212920996b8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:11.451026 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-a8710991-8fcc-4789-a440-2212920996b8 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:11.492467 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fb98c47-5a6e-40cd-ac7f-3b82fa61dd72 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:11.496995 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c93b4dd1-56a2-4077-8a8e-c9a22a3f4f62 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:11.497594 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c93b4dd1-56a2-4077-8a8e-c9a22a3f4f62 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:11.497888 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c93b4dd1-56a2-4077-8a8e-c9a22a3f4f62 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:11.498147 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c93b4dd1-56a2-4077-8a8e-c9a22a3f4f62 admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:11.498769 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c93b4dd1-56a2-4077-8a8e-c9a22a3f4f62 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:11.500738 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c93b4dd1-56a2-4077-8a8e-c9a22a3f4f62 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:11.509050 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:11.511267 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8710991-8fcc-4789-a440-2212920996b8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:11.512759 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8710991-8fcc-4789-a440-2212920996b8 service cinder] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:11.522431 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 994/2014] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:57:11 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 185 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:57:11.523402 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8710991-8fcc-4789-a440-2212920996b8 service cinder] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:11.523558 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1021/2015] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:57:11 2022] POST /identity/v3/users => generated 300 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:57:11.526541 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c93b4dd1-56a2-4077-8a8e-c9a22a3f4f62 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:11.535873 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c93b4dd1-56a2-4077-8a8e-c9a22a3f4f62 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:11.540333 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8710991-8fcc-4789-a440-2212920996b8 service cinder] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:11.540887 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e209880-d04e-4342-b281-95daf9e2fb2b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:11.541633 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e209880-d04e-4342-b281-95daf9e2fb2b admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:11.541915 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e209880-d04e-4342-b281-95daf9e2fb2b admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:11.542182 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e209880-d04e-4342-b281-95daf9e2fb2b admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:11.542921 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e209880-d04e-4342-b281-95daf9e2fb2b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:11.544760 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e209880-d04e-4342-b281-95daf9e2fb2b admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:11.546253 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d86ff916-9990-4b2c-b846-b147a0c0bf87 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:11.546858 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d86ff916-9990-4b2c-b846-b147a0c0bf87 admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:11.547147 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d86ff916-9990-4b2c-b846-b147a0c0bf87 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:11.547424 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d86ff916-9990-4b2c-b846-b147a0c0bf87 admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:11.548104 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d86ff916-9990-4b2c-b846-b147a0c0bf87 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:11.550221 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d86ff916-9990-4b2c-b846-b147a0c0bf87 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:11.559500 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1022/2016] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:57:11 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:57:11.560420 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e209880-d04e-4342-b281-95daf9e2fb2b admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:11.563325 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 995/2017] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:57:11 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:57:11.564190 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d86ff916-9990-4b2c-b846-b147a0c0bf87 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:11.572152 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e209880-d04e-4342-b281-95daf9e2fb2b admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:11.575526 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d86ff916-9990-4b2c-b846-b147a0c0bf87 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:11.577771 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21ad712c-9a32-4200-84c6-10364f2d8586 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:11.577902 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ad712c-9a32-4200-84c6-10364f2d8586 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:11.578143 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ad712c-9a32-4200-84c6-10364f2d8586 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:11.578418 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ad712c-9a32-4200-84c6-10364f2d8586 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:11.579102 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21ad712c-9a32-4200-84c6-10364f2d8586 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:11.580308 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-895dde23-e046-4488-89a9-5c2638a53d3b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:11.580683 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21ad712c-9a32-4200-84c6-10364f2d8586 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:11.581059 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-895dde23-e046-4488-89a9-5c2638a53d3b admin admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:11.581371 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-895dde23-e046-4488-89a9-5c2638a53d3b admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:11.581630 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-895dde23-e046-4488-89a9-5c2638a53d3b admin admin] PATH_INFO: `/v3/projects/9f95152a78a1496ca4266c3255a3e533/users/146df20a8829481f8e6f35708e3debb8/roles/092036ee46ac427baf4d56468575f697` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:11.584387 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-895dde23-e046-4488-89a9-5c2638a53d3b admin admin] RBAC: Authorizing `identity:create_grant(project_id=9f95152a78a1496ca4266c3255a3e533, user_id=146df20a8829481f8e6f35708e3debb8, role_id=092036ee46ac427baf4d56468575f697)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:11.587629 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-895dde23-e046-4488-89a9-5c2638a53d3b admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:11.592490 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1023/2018] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:57:11 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:57:11.593321 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21ad712c-9a32-4200-84c6-10364f2d8586 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:11.600001 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 996/2019] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:57:11 2022] PUT /identity/v3/projects/9f95152a78a1496ca4266c3255a3e533/users/146df20a8829481f8e6f35708e3debb8/roles/092036ee46ac427baf4d56468575f697 => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:11.601119 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-895dde23-e046-4488-89a9-5c2638a53d3b admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:11.616669 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21ad712c-9a32-4200-84c6-10364f2d8586 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:11.676575 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-895dde23-e046-4488-89a9-5c2638a53d3b admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:11.682003 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1465cecc-fc6f-4a66-92a3-43a20635459a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:11.682690 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1465cecc-fc6f-4a66-92a3-43a20635459a admin admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:11.682965 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1465cecc-fc6f-4a66-92a3-43a20635459a admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:11.683236 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1465cecc-fc6f-4a66-92a3-43a20635459a admin admin] PATH_INFO: `/v3/projects/30f25c4e8f0046cfac8a8398ffa075e3/users/73876df946ab4e7ba929fd23f518b887/roles/092036ee46ac427baf4d56468575f697` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:11.689065 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebd0dfa4-b4d0-48e7-9857-fc5d06eaa723 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:11.689745 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd0dfa4-b4d0-48e7-9857-fc5d06eaa723 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:11.690391 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd0dfa4-b4d0-48e7-9857-fc5d06eaa723 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:11.690712 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd0dfa4-b4d0-48e7-9857-fc5d06eaa723 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:11.691642 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebd0dfa4-b4d0-48e7-9857-fc5d06eaa723 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:11.693503 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebd0dfa4-b4d0-48e7-9857-fc5d06eaa723 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:11.701055 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1465cecc-fc6f-4a66-92a3-43a20635459a admin admin] RBAC: Authorizing `identity:create_grant(project_id=30f25c4e8f0046cfac8a8398ffa075e3, user_id=73876df946ab4e7ba929fd23f518b887, role_id=092036ee46ac427baf4d56468575f697)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:11.703784 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1465cecc-fc6f-4a66-92a3-43a20635459a admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:11.705173 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1024/2020] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:57:11 2022] GET /identity/v3/roles => generated 1420 bytes in 112 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:57:11.706955 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c8ed99-620d-4397-b2a1-3fc1f6d7e463 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:11.707238 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c8ed99-620d-4397-b2a1-3fc1f6d7e463 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:11.709175 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c8ed99-620d-4397-b2a1-3fc1f6d7e463 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:11.730879 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 997/2021] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:57:11 2022] PUT /identity/v3/projects/30f25c4e8f0046cfac8a8398ffa075e3/users/73876df946ab4e7ba929fd23f518b887/roles/092036ee46ac427baf4d56468575f697 => generated 0 bytes in 130 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:11.732263 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1465cecc-fc6f-4a66-92a3-43a20635459a admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:11.747246 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-c9c8ed99-620d-4397-b2a1-3fc1f6d7e463 None None] MFA Rules not processed for user `146df20a8829481f8e6f35708e3debb8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:11.748970 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-c9c8ed99-620d-4397-b2a1-3fc1f6d7e463 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:11.809887 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:11.824913 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1465cecc-fc6f-4a66-92a3-43a20635459a admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:11.837652 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1025/2022] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:11 2022] POST /identity/v3/auth/tokens => generated 3333 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:57:11.838759 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbf528ce-4adb-499a-a1f1-bd078a6e5c27 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:11.839245 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9c8ed99-620d-4397-b2a1-3fc1f6d7e463 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:11.839439 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf528ce-4adb-499a-a1f1-bd078a6e5c27 admin admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:11.839756 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf528ce-4adb-499a-a1f1-bd078a6e5c27 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:11.840032 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf528ce-4adb-499a-a1f1-bd078a6e5c27 admin admin] PATH_INFO: `/v3/projects/aaf669b496984e38b2d5621c650ab60d/users/cd80a4e0abf841c79e6b537410c5cb00/roles/588016933bab436d8b7c3ac76827356d` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:11.859294 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9c8ed99-620d-4397-b2a1-3fc1f6d7e463 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:11.866280 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbf528ce-4adb-499a-a1f1-bd078a6e5c27 admin admin] RBAC: Authorizing `identity:create_grant(project_id=aaf669b496984e38b2d5621c650ab60d, user_id=cd80a4e0abf841c79e6b537410c5cb00, role_id=588016933bab436d8b7c3ac76827356d)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:11.869564 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbf528ce-4adb-499a-a1f1-bd078a6e5c27 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:11.869705 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f34886c-fb8e-49f9-bf23-e9e2cb768ea9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:11.870108 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f34886c-fb8e-49f9-bf23-e9e2cb768ea9 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:11.870556 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f34886c-fb8e-49f9-bf23-e9e2cb768ea9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:11.870845 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f34886c-fb8e-49f9-bf23-e9e2cb768ea9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:11.871483 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f34886c-fb8e-49f9-bf23-e9e2cb768ea9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:11.874182 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f34886c-fb8e-49f9-bf23-e9e2cb768ea9 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:11.885075 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1026/2023] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:57:11 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:57:11.885075 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f34886c-fb8e-49f9-bf23-e9e2cb768ea9 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:11.889923 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-3f34886c-fb8e-49f9-bf23-e9e2cb768ea9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:11.891524 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 998/2024] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:57:11 2022] PUT /identity/v3/projects/aaf669b496984e38b2d5621c650ab60d/users/cd80a4e0abf841c79e6b537410c5cb00/roles/588016933bab436d8b7c3ac76827356d => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:11.900933 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbf528ce-4adb-499a-a1f1-bd078a6e5c27 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:11.948960 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:11.977047 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f34886c-fb8e-49f9-bf23-e9e2cb768ea9 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:11.980267 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbf528ce-4adb-499a-a1f1-bd078a6e5c27 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:11.987604 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8d7d85d-9d5c-43b5-b416-152021af9994 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:11.988286 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8d7d85d-9d5c-43b5-b416-152021af9994 admin admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:11.988565 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8d7d85d-9d5c-43b5-b416-152021af9994 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:11.988845 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8d7d85d-9d5c-43b5-b416-152021af9994 admin admin] PATH_INFO: `/v3/projects/30f25c4e8f0046cfac8a8398ffa075e3/users/73876df946ab4e7ba929fd23f518b887/roles/092036ee46ac427baf4d56468575f697` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:11.991025 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8d7d85d-9d5c-43b5-b416-152021af9994 admin admin] RBAC: Authorizing `identity:create_grant(project_id=30f25c4e8f0046cfac8a8398ffa075e3, user_id=73876df946ab4e7ba929fd23f518b887, role_id=092036ee46ac427baf4d56468575f697)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:11.994030 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8d7d85d-9d5c-43b5-b416-152021af9994 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:12.014490 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 999/2025] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:57:11 2022] PUT /identity/v3/projects/30f25c4e8f0046cfac8a8398ffa075e3/users/73876df946ab4e7ba929fd23f518b887/roles/092036ee46ac427baf4d56468575f697 => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:12.016881 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de112142-4cd1-45cc-a862-917b34e3bf84 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:12.017226 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de112142-4cd1-45cc-a862-917b34e3bf84 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:12.017749 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de112142-4cd1-45cc-a862-917b34e3bf84 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:12.040825 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cc280bf-89d8-4396-8a54-4279182d399c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3f48335668a4e8f934ac11ff66cd308', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:12.042091 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc280bf-89d8-4396-8a54-4279182d399c service cinder] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:12.042582 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc280bf-89d8-4396-8a54-4279182d399c service cinder] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:12.042949 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc280bf-89d8-4396-8a54-4279182d399c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:12.050943 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-de112142-4cd1-45cc-a862-917b34e3bf84 None None] MFA Rules not processed for user `cd80a4e0abf841c79e6b537410c5cb00`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:12.052214 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-de112142-4cd1-45cc-a862-917b34e3bf84 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:12.122717 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cc280bf-89d8-4396-8a54-4279182d399c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:12.124340 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cc280bf-89d8-4396-8a54-4279182d399c service cinder] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:12.136190 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1027/2026] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:57:11 2022] GET /identity/v3/auth/tokens => generated 3333 bytes in 251 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:57:12.137587 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a58949-371b-4f7d-a88a-2a786b07e469 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:12.137944 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a58949-371b-4f7d-a88a-2a786b07e469 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:12.138266 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a58949-371b-4f7d-a88a-2a786b07e469 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:12.147344 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1000/2027] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:12 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:57:12.148530 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de112142-4cd1-45cc-a862-917b34e3bf84 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:12.169928 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-37a58949-371b-4f7d-a88a-2a786b07e469 None None] MFA Rules not processed for user `73876df946ab4e7ba929fd23f518b887`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:12.171204 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-37a58949-371b-4f7d-a88a-2a786b07e469 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:12.208305 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:12.233111 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de112142-4cd1-45cc-a862-917b34e3bf84 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:12.241259 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc18c331-e8ee-4a9b-b7f1-68e8cc26fd98 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:12.242073 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc18c331-e8ee-4a9b-b7f1-68e8cc26fd98 admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:12.242325 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc18c331-e8ee-4a9b-b7f1-68e8cc26fd98 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:12.242679 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc18c331-e8ee-4a9b-b7f1-68e8cc26fd98 admin admin] PATH_INFO: `/v3/projects` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:12.243806 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc18c331-e8ee-4a9b-b7f1-68e8cc26fd98 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:12.248273 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc18c331-e8ee-4a9b-b7f1-68e8cc26fd98 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:12.263629 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1028/2028] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:12 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:57:12.276178 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37a58949-371b-4f7d-a88a-2a786b07e469 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:12.286242 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1001/2029] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 00:57:12 2022] POST /identity/v3/projects => generated 379 bytes in 135 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:57:12.288723 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc18c331-e8ee-4a9b-b7f1-68e8cc26fd98 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:12.339647 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37a58949-371b-4f7d-a88a-2a786b07e469 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:12.354655 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:12.378629 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc18c331-e8ee-4a9b-b7f1-68e8cc26fd98 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:12.384243 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f41ef6b-0e9c-4b41-a2c6-3993ba7bb45d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:12.384920 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f41ef6b-0e9c-4b41-a2c6-3993ba7bb45d admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:12.385201 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f41ef6b-0e9c-4b41-a2c6-3993ba7bb45d admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:12.385473 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f41ef6b-0e9c-4b41-a2c6-3993ba7bb45d admin admin] PATH_INFO: `/v3/users` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:12.386100 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f41ef6b-0e9c-4b41-a2c6-3993ba7bb45d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:12.388074 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f41ef6b-0e9c-4b41-a2c6-3993ba7bb45d admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:12.400104 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0be2f4dc-e052-4135-9893-1d62edcacf1a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3f48335668a4e8f934ac11ff66cd308', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:12.401029 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be2f4dc-e052-4135-9893-1d62edcacf1a service cinder] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:12.401386 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be2f4dc-e052-4135-9893-1d62edcacf1a service cinder] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:12.401809 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be2f4dc-e052-4135-9893-1d62edcacf1a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:12.412204 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1002/2030] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:57:12 2022] POST /identity/v3/users => generated 290 bytes in 124 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:57:12.412985 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f41ef6b-0e9c-4b41-a2c6-3993ba7bb45d admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:12.427890 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f41ef6b-0e9c-4b41-a2c6-3993ba7bb45d admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:12.433411 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c923091-208c-48f9-8513-e23065fd74c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:12.434410 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c923091-208c-48f9-8513-e23065fd74c3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:12.434842 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c923091-208c-48f9-8513-e23065fd74c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:12.434997 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c923091-208c-48f9-8513-e23065fd74c3 admin admin] PATH_INFO: `/v3/users/785b47656c7747328d3e6235f86968a4` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:12.436341 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c923091-208c-48f9-8513-e23065fd74c3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=785b47656c7747328d3e6235f86968a4)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:12.438275 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c923091-208c-48f9-8513-e23065fd74c3 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:12.486907 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0be2f4dc-e052-4135-9893-1d62edcacf1a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:12.489053 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0be2f4dc-e052-4135-9893-1d62edcacf1a service cinder] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:12.497892 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-5c923091-208c-48f9-8513-e23065fd74c3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '785b47656c7747328d3e6235f86968a4'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:12.499075 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1029/2031] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:57:12 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 223 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 09 00:57:12.499905 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0be2f4dc-e052-4135-9893-1d62edcacf1a service cinder] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:12.508125 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-5c923091-208c-48f9-8513-e23065fd74c3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '785b47656c7747328d3e6235f86968a4'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:12.508842 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-5c923091-208c-48f9-8513-e23065fd74c3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '785b47656c7747328d3e6235f86968a4'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:12.510165 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0be2f4dc-e052-4135-9893-1d62edcacf1a service cinder] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:12.515202 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64069e18-ab5a-471d-b679-eab341990e63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:12.515771 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64069e18-ab5a-471d-b679-eab341990e63 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:12.516088 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64069e18-ab5a-471d-b679-eab341990e63 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:12.516453 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64069e18-ab5a-471d-b679-eab341990e63 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:12.517019 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64069e18-ab5a-471d-b679-eab341990e63 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:12.518472 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-5c923091-208c-48f9-8513-e23065fd74c3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '785b47656c7747328d3e6235f86968a4'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:12.518686 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64069e18-ab5a-471d-b679-eab341990e63 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:12.533149 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1030/2032] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:57:12 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:57:12.534203 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64069e18-ab5a-471d-b679-eab341990e63 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:12.536019 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-64069e18-ab5a-471d-b679-eab341990e63 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:12.542820 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-5c923091-208c-48f9-8513-e23065fd74c3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '785b47656c7747328d3e6235f86968a4'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:12.550600 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1003/2033] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 00:57:12 2022] DELETE /identity/v3/users/785b47656c7747328d3e6235f86968a4 => generated 0 bytes in 137 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:12.550767 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64069e18-ab5a-471d-b679-eab341990e63 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:12.554231 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9e2022-32ec-4de9-8d6c-89165e367d06 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:12.554525 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9e2022-32ec-4de9-8d6c-89165e367d06 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:12.554806 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9e2022-32ec-4de9-8d6c-89165e367d06 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:12.572423 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:12.582953 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-5b9e2022-32ec-4de9-8d6c-89165e367d06 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:12.584211 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-5b9e2022-32ec-4de9-8d6c-89165e367d06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:12.618040 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77803967-1e4e-496f-8031-a5d0865e3f31 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:12.618627 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77803967-1e4e-496f-8031-a5d0865e3f31 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:12.618904 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77803967-1e4e-496f-8031-a5d0865e3f31 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:12.619175 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77803967-1e4e-496f-8031-a5d0865e3f31 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:12.619858 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77803967-1e4e-496f-8031-a5d0865e3f31 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:12.621559 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77803967-1e4e-496f-8031-a5d0865e3f31 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:12.632290 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1031/2034] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:57:12 2022] GET /identity/v3/roles => generated 1420 bytes in 98 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 09 00:57:12.636363 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77803967-1e4e-496f-8031-a5d0865e3f31 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:12.648589 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77803967-1e4e-496f-8031-a5d0865e3f31 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:12.654210 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cfc3140-d6a0-4f22-aa39-9ab179f1058d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:12.654900 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cfc3140-d6a0-4f22-aa39-9ab179f1058d admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:12.655183 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cfc3140-d6a0-4f22-aa39-9ab179f1058d admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:12.655458 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cfc3140-d6a0-4f22-aa39-9ab179f1058d admin admin] PATH_INFO: `/v3/projects/93fd69ab23c54fb08cff7e8d89e7adbf/users/114bcf4dd21e4f64980c9e41cfb4dab5/roles/588016933bab436d8b7c3ac76827356d` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:12.661305 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1004/2035] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:57:12.671236 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b9e2022-32ec-4de9-8d6c-89165e367d06 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:12.672967 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cfc3140-d6a0-4f22-aa39-9ab179f1058d admin admin] RBAC: Authorizing `identity:create_grant(project_id=93fd69ab23c54fb08cff7e8d89e7adbf, user_id=114bcf4dd21e4f64980c9e41cfb4dab5, role_id=588016933bab436d8b7c3ac76827356d)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:12.673409 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-5b9e2022-32ec-4de9-8d6c-89165e367d06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:12.676671 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cfc3140-d6a0-4f22-aa39-9ab179f1058d admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:12.701017 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1032/2036] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:57:12 2022] PUT /identity/v3/projects/93fd69ab23c54fb08cff7e8d89e7adbf/users/114bcf4dd21e4f64980c9e41cfb4dab5/roles/588016933bab436d8b7c3ac76827356d => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 09 00:57:12.705884 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72dfdf16-a7ad-4207-9c95-a19af746668c None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:12.706246 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72dfdf16-a7ad-4207-9c95-a19af746668c None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:12.706814 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72dfdf16-a7ad-4207-9c95-a19af746668c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:12.738309 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-72dfdf16-a7ad-4207-9c95-a19af746668c None None] MFA Rules not processed for user `114bcf4dd21e4f64980c9e41cfb4dab5`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:12.739517 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-72dfdf16-a7ad-4207-9c95-a19af746668c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:12.755255 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=86,keystone:INSERT=4,keystone:DELETE=10 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:12.759296 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b9e2022-32ec-4de9-8d6c-89165e367d06 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:12.821649 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1033/2037] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:12 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 09 00:57:12.824528 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4b197cb-8e3a-4ab9-a8da-3af96d920dfd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:12.825461 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4b197cb-8e3a-4ab9-a8da-3af96d920dfd service neutron] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:12.825756 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4b197cb-8e3a-4ab9-a8da-3af96d920dfd service neutron] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:12.826045 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4b197cb-8e3a-4ab9-a8da-3af96d920dfd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:12.826686 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72dfdf16-a7ad-4207-9c95-a19af746668c None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:12.910526 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72dfdf16-a7ad-4207-9c95-a19af746668c None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:12.913761 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4b197cb-8e3a-4ab9-a8da-3af96d920dfd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:12.915200 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4b197cb-8e3a-4ab9-a8da-3af96d920dfd service neutron] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:12.916117 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cf6d334-3383-46af-b7ec-5ce3deb1667f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:12.917091 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf6d334-3383-46af-b7ec-5ce3deb1667f admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:12.917253 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf6d334-3383-46af-b7ec-5ce3deb1667f admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:12.917674 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf6d334-3383-46af-b7ec-5ce3deb1667f admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:12.920090 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cf6d334-3383-46af-b7ec-5ce3deb1667f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:12.935100 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cf6d334-3383-46af-b7ec-5ce3deb1667f admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:12.940386 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1005/2038] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:57:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 270 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 09 00:57:12.976069 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1034/2039] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 00:57:12 2022] POST /identity/v3/projects => generated 379 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 09 00:57:12.979986 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4b197cb-8e3a-4ab9-a8da-3af96d920dfd service neutron] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:12.993940 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4b197cb-8e3a-4ab9-a8da-3af96d920dfd service neutron] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:13.105880 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca53d123-4769-4b08-b0e1-0308f3c045bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:13.106757 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca53d123-4769-4b08-b0e1-0308f3c045bd admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:13.107107 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca53d123-4769-4b08-b0e1-0308f3c045bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:13.107454 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca53d123-4769-4b08-b0e1-0308f3c045bd admin admin] PATH_INFO: `/v3/users` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:13.108543 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca53d123-4769-4b08-b0e1-0308f3c045bd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:13.110769 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca53d123-4769-4b08-b0e1-0308f3c045bd admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:13.158462 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1006/2040] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:57:12 2022] POST /identity/v3/users => generated 296 bytes in 175 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 09 00:57:13.160414 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cf6d334-3383-46af-b7ec-5ce3deb1667f admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:13.172434 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cf6d334-3383-46af-b7ec-5ce3deb1667f admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:13.179177 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e2c555e-99a3-4005-9124-9b37d577a92f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:13.179905 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e2c555e-99a3-4005-9124-9b37d577a92f admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:13.180253 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e2c555e-99a3-4005-9124-9b37d577a92f admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:13.180584 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e2c555e-99a3-4005-9124-9b37d577a92f admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:13.181621 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e2c555e-99a3-4005-9124-9b37d577a92f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:13.183625 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e2c555e-99a3-4005-9124-9b37d577a92f admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:13.186955 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:13.203187 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1035/2041] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:57:13 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:57:13.203939 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca53d123-4769-4b08-b0e1-0308f3c045bd admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:13.226845 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca53d123-4769-4b08-b0e1-0308f3c045bd admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:13.226845 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f5921c-eb52-4561-bebf-68eb0d949fb0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:13.226845 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f5921c-eb52-4561-bebf-68eb0d949fb0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:13.226845 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f5921c-eb52-4561-bebf-68eb0d949fb0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:13.227124 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f5921c-eb52-4561-bebf-68eb0d949fb0 admin admin] PATH_INFO: `/v3/projects/ea6cd31ea35842bd9e36c7a3b66a1fb3/users/da1827fbf6544a9e877dcf224c873ed1/roles/092036ee46ac427baf4d56468575f697` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:13.243355 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53f5921c-eb52-4561-bebf-68eb0d949fb0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ea6cd31ea35842bd9e36c7a3b66a1fb3, user_id=da1827fbf6544a9e877dcf224c873ed1, role_id=092036ee46ac427baf4d56468575f697)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:13.247033 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53f5921c-eb52-4561-bebf-68eb0d949fb0 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:13.264019 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1007/2042] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:57:13 2022] PUT /identity/v3/projects/ea6cd31ea35842bd9e36c7a3b66a1fb3/users/da1827fbf6544a9e877dcf224c873ed1/roles/092036ee46ac427baf4d56468575f697 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 09 00:57:13.267916 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e2c555e-99a3-4005-9124-9b37d577a92f admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:13.269750 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f5921c-eb52-4561-bebf-68eb0d949fb0 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:13.270596 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-6e2c555e-99a3-4005-9124-9b37d577a92f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:13.285558 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e2c555e-99a3-4005-9124-9b37d577a92f admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:13.300916 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:13.351140 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f5921c-eb52-4561-bebf-68eb0d949fb0 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:13.352720 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c16e95a-ee2d-40be-b99a-84b79c6798e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:13.353354 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c16e95a-ee2d-40be-b99a-84b79c6798e2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:13.353633 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c16e95a-ee2d-40be-b99a-84b79c6798e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:13.353908 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c16e95a-ee2d-40be-b99a-84b79c6798e2 admin admin] PATH_INFO: `/v3/users/9a897077384a4ee8a1c87d12dfb8148c` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:13.355014 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c16e95a-ee2d-40be-b99a-84b79c6798e2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9a897077384a4ee8a1c87d12dfb8148c)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:13.355717 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33673141-6584-49d0-b04f-2f6f0b177561 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:13.356313 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33673141-6584-49d0-b04f-2f6f0b177561 admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:13.356587 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33673141-6584-49d0-b04f-2f6f0b177561 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:13.356855 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33673141-6584-49d0-b04f-2f6f0b177561 admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:13.356946 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c16e95a-ee2d-40be-b99a-84b79c6798e2 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:13.357486 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33673141-6584-49d0-b04f-2f6f0b177561 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:13.359043 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33673141-6584-49d0-b04f-2f6f0b177561 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:13.368855 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1008/2043] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:57:13 2022] GET /identity/v3/roles => generated 1420 bytes in 100 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:57:13.369743 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33673141-6584-49d0-b04f-2f6f0b177561 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:13.371402 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-33673141-6584-49d0-b04f-2f6f0b177561 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:13.385192 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33673141-6584-49d0-b04f-2f6f0b177561 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:13.390242 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0a3122c-d577-47d1-a490-852737a0c8f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:13.390871 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0a3122c-d577-47d1-a490-852737a0c8f7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:13.391180 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0a3122c-d577-47d1-a490-852737a0c8f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:13.391454 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0a3122c-d577-47d1-a490-852737a0c8f7 admin admin] PATH_INFO: `/v3/projects/88ea5df9d105498d896615626269e56a` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:13.392723 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0a3122c-d577-47d1-a490-852737a0c8f7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=88ea5df9d105498d896615626269e56a)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:13.395106 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0a3122c-d577-47d1-a490-852737a0c8f7 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:13.404512 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-0c16e95a-ee2d-40be-b99a-84b79c6798e2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9a897077384a4ee8a1c87d12dfb8148c'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:13.412308 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-0c16e95a-ee2d-40be-b99a-84b79c6798e2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9a897077384a4ee8a1c87d12dfb8148c'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:13.412966 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-0c16e95a-ee2d-40be-b99a-84b79c6798e2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9a897077384a4ee8a1c87d12dfb8148c'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:13.413590 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-0c16e95a-ee2d-40be-b99a-84b79c6798e2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9a897077384a4ee8a1c87d12dfb8148c'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:13.420636 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-0c16e95a-ee2d-40be-b99a-84b79c6798e2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9a897077384a4ee8a1c87d12dfb8148c'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:13.439246 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1036/2044] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 00:57:13 2022] DELETE /identity/v3/users/9a897077384a4ee8a1c87d12dfb8148c => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:13.440217 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c16e95a-ee2d-40be-b99a-84b79c6798e2 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:13.441752 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-0c16e95a-ee2d-40be-b99a-84b79c6798e2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:13.442153 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-e0a3122c-d577-47d1-a490-852737a0c8f7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '88ea5df9d105498d896615626269e56a', 'request_id': 'req-e0a3122c-d577-47d1-a490-852737a0c8f7'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:13.448210 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-e0a3122c-d577-47d1-a490-852737a0c8f7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '88ea5df9d105498d896615626269e56a', 'request_id': 'req-e0a3122c-d577-47d1-a490-852737a0c8f7'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:13.455130 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-e0a3122c-d577-47d1-a490-852737a0c8f7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '88ea5df9d105498d896615626269e56a', 'request_id': 'req-e0a3122c-d577-47d1-a490-852737a0c8f7'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:13.467933 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-e0a3122c-d577-47d1-a490-852737a0c8f7 admin admin] The token cache is being invalidate because project 88ea5df9d105498d896615626269e56a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81453) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 00:57:13.468232 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-e0a3122c-d577-47d1-a490-852737a0c8f7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:13.468872 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-e0a3122c-d577-47d1-a490-852737a0c8f7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:13.470634 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1009/2045] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 00:57:13 2022] DELETE /identity/v3/projects/88ea5df9d105498d896615626269e56a => generated 0 bytes in 101 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:13.470634 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-159464ca-7de6-4380-9079-4ffc0b0db686 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:13.470634 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-159464ca-7de6-4380-9079-4ffc0b0db686 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:13.470634 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-159464ca-7de6-4380-9079-4ffc0b0db686 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:13.504477 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-159464ca-7de6-4380-9079-4ffc0b0db686 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:13.504477 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-159464ca-7de6-4380-9079-4ffc0b0db686 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:13.510331 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c16e95a-ee2d-40be-b99a-84b79c6798e2 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:13.511661 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1010/2046] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:57:13.512423 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-159464ca-7de6-4380-9079-4ffc0b0db686 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:13.513760 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-159464ca-7de6-4380-9079-4ffc0b0db686 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:13.521369 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48b3d964-142c-4524-b786-3a2b740d2075 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:13.522112 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48b3d964-142c-4524-b786-3a2b740d2075 admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:13.522521 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48b3d964-142c-4524-b786-3a2b740d2075 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:13.522872 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48b3d964-142c-4524-b786-3a2b740d2075 admin admin] PATH_INFO: `/v3/projects/ea6cd31ea35842bd9e36c7a3b66a1fb3/users/da1827fbf6544a9e877dcf224c873ed1/roles/092036ee46ac427baf4d56468575f697` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:13.524934 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48b3d964-142c-4524-b786-3a2b740d2075 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ea6cd31ea35842bd9e36c7a3b66a1fb3, user_id=da1827fbf6544a9e877dcf224c873ed1, role_id=092036ee46ac427baf4d56468575f697)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:13.526785 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-159464ca-7de6-4380-9079-4ffc0b0db686 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:13.527771 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48b3d964-142c-4524-b786-3a2b740d2075 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:13.532398 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a8bcf7d-8d3d-4a8d-a805-743eeca63333 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:13.533035 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a8bcf7d-8d3d-4a8d-a805-743eeca63333 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:13.533330 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a8bcf7d-8d3d-4a8d-a805-743eeca63333 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:13.533566 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a8bcf7d-8d3d-4a8d-a805-743eeca63333 admin admin] PATH_INFO: `/v3/users/146df20a8829481f8e6f35708e3debb8` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:13.535326 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a8bcf7d-8d3d-4a8d-a805-743eeca63333 admin admin] RBAC: Authorizing `identity:delete_user(user_id=146df20a8829481f8e6f35708e3debb8)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:13.537515 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a8bcf7d-8d3d-4a8d-a805-743eeca63333 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:13.539793 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1037/2047] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:57:13 2022] PUT /identity/v3/projects/ea6cd31ea35842bd9e36c7a3b66a1fb3/users/da1827fbf6544a9e877dcf224c873ed1/roles/092036ee46ac427baf4d56468575f697 => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:13.540693 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48b3d964-142c-4524-b786-3a2b740d2075 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:13.542459 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-48b3d964-142c-4524-b786-3a2b740d2075 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:13.566496 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:13.604178 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-4a8bcf7d-8d3d-4a8d-a805-743eeca63333 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '146df20a8829481f8e6f35708e3debb8'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:13.611529 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-4a8bcf7d-8d3d-4a8d-a805-743eeca63333 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '146df20a8829481f8e6f35708e3debb8'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:13.612375 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-4a8bcf7d-8d3d-4a8d-a805-743eeca63333 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '146df20a8829481f8e6f35708e3debb8'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:13.619006 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48b3d964-142c-4524-b786-3a2b740d2075 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:13.621937 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-4a8bcf7d-8d3d-4a8d-a805-743eeca63333 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '146df20a8829481f8e6f35708e3debb8'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:13.623095 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f903f624-7653-49a4-901f-f313ef06670f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:13.623773 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f903f624-7653-49a4-901f-f313ef06670f service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:13.624080 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f903f624-7653-49a4-901f-f313ef06670f service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:13.624333 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f903f624-7653-49a4-901f-f313ef06670f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:13.626264 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-f903f624-7653-49a4-901f-f313ef06670f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:13.635881 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-4a8bcf7d-8d3d-4a8d-a805-743eeca63333 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '146df20a8829481f8e6f35708e3debb8'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:13.644099 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1011/2048] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 00:57:13 2022] DELETE /identity/v3/users/146df20a8829481f8e6f35708e3debb8 => generated 0 bytes in 131 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:13.645478 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9598039-357b-4aa9-8168-65fda5642237 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:13.645778 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9598039-357b-4aa9-8168-65fda5642237 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:13.646040 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9598039-357b-4aa9-8168-65fda5642237 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:13.654193 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:13.680369 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-d9598039-357b-4aa9-8168-65fda5642237 None None] MFA Rules not processed for user `da1827fbf6544a9e877dcf224c873ed1`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:13.682189 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-d9598039-357b-4aa9-8168-65fda5642237 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:13.705094 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f903f624-7653-49a4-901f-f313ef06670f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:13.707102 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f903f624-7653-49a4-901f-f313ef06670f service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:13.719568 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1038/2049] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:57:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:57:13.786009 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1012/2050] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:13 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:57:13.804999 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f903f624-7653-49a4-901f-f313ef06670f service neutron] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:13.807290 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-f903f624-7653-49a4-901f-f313ef06670f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:13.908473 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f903f624-7653-49a4-901f-f313ef06670f service neutron] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:14.026540 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e7a75ff-ec91-48a5-b586-f5a03765ed9c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3f48335668a4e8f934ac11ff66cd308', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:14.027233 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e7a75ff-ec91-48a5-b586-f5a03765ed9c service cinder] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:14.027629 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e7a75ff-ec91-48a5-b586-f5a03765ed9c service cinder] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:14.028107 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e7a75ff-ec91-48a5-b586-f5a03765ed9c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:14.039155 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e7a75ff-ec91-48a5-b586-f5a03765ed9c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:14.040947 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e7a75ff-ec91-48a5-b586-f5a03765ed9c service cinder] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:14.052584 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1039/2051] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:57:13 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 249 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:57:14.061756 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9598039-357b-4aa9-8168-65fda5642237 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:14.063478 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-d9598039-357b-4aa9-8168-65fda5642237 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:14.072458 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e7a75ff-ec91-48a5-b586-f5a03765ed9c service cinder] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:14.086548 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-2e7a75ff-ec91-48a5-b586-f5a03765ed9c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:14.136481 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:14.177610 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9598039-357b-4aa9-8168-65fda5642237 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:14.183040 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e34f1f7-6379-4e64-a965-950b28c8e9d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:14.184569 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e34f1f7-6379-4e64-a965-950b28c8e9d3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:14.184901 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e34f1f7-6379-4e64-a965-950b28c8e9d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:14.185174 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e34f1f7-6379-4e64-a965-950b28c8e9d3 admin admin] PATH_INFO: `/v3/users/3f85d2ae83134254aa1fb11c1e500767` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:14.186485 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e34f1f7-6379-4e64-a965-950b28c8e9d3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3f85d2ae83134254aa1fb11c1e500767)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:14.188468 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e34f1f7-6379-4e64-a965-950b28c8e9d3 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:14.204807 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e7a75ff-ec91-48a5-b586-f5a03765ed9c service cinder] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:14.211749 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e5be243-0530-4cfe-9a0a-aa7188f0af26 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:14.212568 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e5be243-0530-4cfe-9a0a-aa7188f0af26 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:14.212850 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e5be243-0530-4cfe-9a0a-aa7188f0af26 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:14.213115 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e5be243-0530-4cfe-9a0a-aa7188f0af26 admin admin] PATH_INFO: `/v3/projects/9f95152a78a1496ca4266c3255a3e533` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:14.214941 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e5be243-0530-4cfe-9a0a-aa7188f0af26 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9f95152a78a1496ca4266c3255a3e533)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:14.218806 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e5be243-0530-4cfe-9a0a-aa7188f0af26 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:14.266972 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-4e34f1f7-6379-4e64-a965-950b28c8e9d3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3f85d2ae83134254aa1fb11c1e500767'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:14.276990 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-4e34f1f7-6379-4e64-a965-950b28c8e9d3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3f85d2ae83134254aa1fb11c1e500767'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:14.277928 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-4e34f1f7-6379-4e64-a965-950b28c8e9d3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3f85d2ae83134254aa1fb11c1e500767'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:14.289499 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-4e34f1f7-6379-4e64-a965-950b28c8e9d3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3f85d2ae83134254aa1fb11c1e500767'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:14.295084 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-9e5be243-0530-4cfe-9a0a-aa7188f0af26 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9f95152a78a1496ca4266c3255a3e533', 'request_id': 'req-9e5be243-0530-4cfe-9a0a-aa7188f0af26'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:14.301834 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-9e5be243-0530-4cfe-9a0a-aa7188f0af26 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9f95152a78a1496ca4266c3255a3e533', 'request_id': 'req-9e5be243-0530-4cfe-9a0a-aa7188f0af26'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:14.306909 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-4e34f1f7-6379-4e64-a965-950b28c8e9d3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3f85d2ae83134254aa1fb11c1e500767'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:14.308476 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-9e5be243-0530-4cfe-9a0a-aa7188f0af26 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9f95152a78a1496ca4266c3255a3e533', 'request_id': 'req-9e5be243-0530-4cfe-9a0a-aa7188f0af26'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:14.316479 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1013/2052] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 00:57:14 2022] DELETE /identity/v3/users/3f85d2ae83134254aa1fb11c1e500767 => generated 0 bytes in 256 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:14.317667 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e34f1f7-6379-4e64-a965-950b28c8e9d3 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:14.320904 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-4e34f1f7-6379-4e64-a965-950b28c8e9d3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:14.323067 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-9e5be243-0530-4cfe-9a0a-aa7188f0af26 admin admin] The token cache is being invalidate because project 9f95152a78a1496ca4266c3255a3e533 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 00:57:14.323067 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-9e5be243-0530-4cfe-9a0a-aa7188f0af26 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:14.323778 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-9e5be243-0530-4cfe-9a0a-aa7188f0af26 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:14.325335 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1040/2053] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 00:57:14 2022] DELETE /identity/v3/projects/9f95152a78a1496ca4266c3255a3e533 => generated 0 bytes in 254 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:14.326268 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e5be243-0530-4cfe-9a0a-aa7188f0af26 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:14.327797 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-9e5be243-0530-4cfe-9a0a-aa7188f0af26 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:14.336131 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:14.401961 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e34f1f7-6379-4e64-a965-950b28c8e9d3 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:14.403513 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e5be243-0530-4cfe-9a0a-aa7188f0af26 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:14.406098 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f4224c0-9b54-4017-9524-56fe0cf9fcbc service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3f48335668a4e8f934ac11ff66cd308', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:14.406975 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f4224c0-9b54-4017-9524-56fe0cf9fcbc service cinder] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:14.407251 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f4224c0-9b54-4017-9524-56fe0cf9fcbc service cinder] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:14.407522 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f4224c0-9b54-4017-9524-56fe0cf9fcbc service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:14.409886 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-7f4224c0-9b54-4017-9524-56fe0cf9fcbc service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:14.462879 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:14.471308 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed32fdef-8e3c-4749-afd7-61b0c94cc927 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:14.474616 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed32fdef-8e3c-4749-afd7-61b0c94cc927 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:14.475320 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed32fdef-8e3c-4749-afd7-61b0c94cc927 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:14.475671 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed32fdef-8e3c-4749-afd7-61b0c94cc927 admin admin] PATH_INFO: `/v3/projects/fb856bc06ad849b1b275840b69eb18a4` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:14.477282 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed32fdef-8e3c-4749-afd7-61b0c94cc927 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fb856bc06ad849b1b275840b69eb18a4)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:14.478875 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f4224c0-9b54-4017-9524-56fe0cf9fcbc service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:14.480274 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed32fdef-8e3c-4749-afd7-61b0c94cc927 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:14.480702 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f4224c0-9b54-4017-9524-56fe0cf9fcbc service cinder] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:14.492633 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1014/2054] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:57:14 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 175 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:57:14.494446 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7db4747f-8797-4469-add0-e4f42305ad84 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:14.494446 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7db4747f-8797-4469-add0-e4f42305ad84 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:14.494638 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7db4747f-8797-4469-add0-e4f42305ad84 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:14.516284 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-7db4747f-8797-4469-add0-e4f42305ad84 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:14.518372 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-7db4747f-8797-4469-add0-e4f42305ad84 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:14.541336 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-ed32fdef-8e3c-4749-afd7-61b0c94cc927 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fb856bc06ad849b1b275840b69eb18a4', 'request_id': 'req-ed32fdef-8e3c-4749-afd7-61b0c94cc927'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:14.549505 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-ed32fdef-8e3c-4749-afd7-61b0c94cc927 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fb856bc06ad849b1b275840b69eb18a4', 'request_id': 'req-ed32fdef-8e3c-4749-afd7-61b0c94cc927'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:14.558486 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-ed32fdef-8e3c-4749-afd7-61b0c94cc927 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fb856bc06ad849b1b275840b69eb18a4', 'request_id': 'req-ed32fdef-8e3c-4749-afd7-61b0c94cc927'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:14.590002 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-ed32fdef-8e3c-4749-afd7-61b0c94cc927 admin admin] The token cache is being invalidate because project fb856bc06ad849b1b275840b69eb18a4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 00:57:14.590292 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-ed32fdef-8e3c-4749-afd7-61b0c94cc927 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:14.591166 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-ed32fdef-8e3c-4749-afd7-61b0c94cc927 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:14.595675 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1041/2055] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 00:57:14 2022] DELETE /identity/v3/projects/fb856bc06ad849b1b275840b69eb18a4 => generated 0 bytes in 268 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:14.598012 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f33bc24-89e4-4fae-a7db-2e98b5c54868 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:14.598292 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f33bc24-89e4-4fae-a7db-2e98b5c54868 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:14.599206 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f33bc24-89e4-4fae-a7db-2e98b5c54868 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:14.607428 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1015/2056] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:57:14.608402 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7db4747f-8797-4469-add0-e4f42305ad84 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:14.609750 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-7db4747f-8797-4469-add0-e4f42305ad84 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:14.622484 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7db4747f-8797-4469-add0-e4f42305ad84 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:14.628326 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1880111e-a18a-4e57-9c9a-45b90d64139d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:14.629033 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1880111e-a18a-4e57-9c9a-45b90d64139d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:14.629202 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-5f33bc24-89e4-4fae-a7db-2e98b5c54868 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:14.629507 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1880111e-a18a-4e57-9c9a-45b90d64139d admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:14.629765 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1880111e-a18a-4e57-9c9a-45b90d64139d admin admin] PATH_INFO: `/v3/users/73876df946ab4e7ba929fd23f518b887` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:14.630616 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-5f33bc24-89e4-4fae-a7db-2e98b5c54868 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:14.631070 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1880111e-a18a-4e57-9c9a-45b90d64139d admin admin] RBAC: Authorizing `identity:delete_user(user_id=73876df946ab4e7ba929fd23f518b887)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:14.633090 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1880111e-a18a-4e57-9c9a-45b90d64139d admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:14.640485 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1042/2057] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:57:14.641366 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f33bc24-89e4-4fae-a7db-2e98b5c54868 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:14.642924 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-5f33bc24-89e4-4fae-a7db-2e98b5c54868 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:14.699055 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-1880111e-a18a-4e57-9c9a-45b90d64139d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '73876df946ab4e7ba929fd23f518b887'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:14.708439 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-1880111e-a18a-4e57-9c9a-45b90d64139d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '73876df946ab4e7ba929fd23f518b887'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:14.709561 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-1880111e-a18a-4e57-9c9a-45b90d64139d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '73876df946ab4e7ba929fd23f518b887'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:14.714339 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:14.715516 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f33bc24-89e4-4fae-a7db-2e98b5c54868 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:14.719773 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96fa7dfe-de54-4e67-8384-e078854f9b04 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:14.720398 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96fa7dfe-de54-4e67-8384-e078854f9b04 service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:14.720672 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96fa7dfe-de54-4e67-8384-e078854f9b04 service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:14.721014 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96fa7dfe-de54-4e67-8384-e078854f9b04 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:14.722792 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-1880111e-a18a-4e57-9c9a-45b90d64139d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '73876df946ab4e7ba929fd23f518b887'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:14.723111 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-96fa7dfe-de54-4e67-8384-e078854f9b04 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:14.738600 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96fa7dfe-de54-4e67-8384-e078854f9b04 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:14.740248 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96fa7dfe-de54-4e67-8384-e078854f9b04 service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:14.740791 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-1880111e-a18a-4e57-9c9a-45b90d64139d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '73876df946ab4e7ba929fd23f518b887'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:14.747735 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1016/2058] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 00:57:14 2022] DELETE /identity/v3/users/73876df946ab4e7ba929fd23f518b887 => generated 0 bytes in 140 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:14.749119 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f43ee45d-ed46-4629-9297-9a87d090eb89 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:14.749453 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f43ee45d-ed46-4629-9297-9a87d090eb89 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:14.750055 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f43ee45d-ed46-4629-9297-9a87d090eb89 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:14.754934 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1043/2059] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:57:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 114 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:57:14.772817 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-f43ee45d-ed46-4629-9297-9a87d090eb89 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:14.774255 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-f43ee45d-ed46-4629-9297-9a87d090eb89 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:14.858206 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1017/2060] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:57:14.864322 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96fa7dfe-de54-4e67-8384-e078854f9b04 service neutron] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:14.875076 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96fa7dfe-de54-4e67-8384-e078854f9b04 service neutron] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:14.889078 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a59d2539-7545-497e-bcb2-acb695b840e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:14.889948 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a59d2539-7545-497e-bcb2-acb695b840e7 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:14.890239 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a59d2539-7545-497e-bcb2-acb695b840e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:14.890518 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a59d2539-7545-497e-bcb2-acb695b840e7 admin admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:14.891171 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a59d2539-7545-497e-bcb2-acb695b840e7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:57:14.891474 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a59d2539-7545-497e-bcb2-acb695b840e7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:14.893307 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a59d2539-7545-497e-bcb2-acb695b840e7 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:14.914604 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1044/2061] 10.222.0.33 () {58 vars in 1153 bytes} [Sun Oct 9 00:57:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:57:14.920562 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f43ee45d-ed46-4629-9297-9a87d090eb89 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:14.934842 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f43ee45d-ed46-4629-9297-9a87d090eb89 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:14.954882 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d10267b4-b7c8-4d8d-9046-aaba21bd9098 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:14.954882 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10267b4-b7c8-4d8d-9046-aaba21bd9098 admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:14.954882 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10267b4-b7c8-4d8d-9046-aaba21bd9098 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:14.954882 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10267b4-b7c8-4d8d-9046-aaba21bd9098 admin admin] PATH_INFO: `/v3/projects` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:14.955202 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d10267b4-b7c8-4d8d-9046-aaba21bd9098 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:14.955202 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d10267b4-b7c8-4d8d-9046-aaba21bd9098 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:14.991648 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1018/2062] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 00:57:14 2022] POST /identity/v3/projects => generated 373 bytes in 70 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:57:14.994798 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a59d2539-7545-497e-bcb2-acb695b840e7 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:15.006068 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a59d2539-7545-497e-bcb2-acb695b840e7 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:15.100415 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b4fd2a7-ab0a-4fc0-9d42-d5f6ca1bdb62 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:15.101494 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b4fd2a7-ab0a-4fc0-9d42-d5f6ca1bdb62 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:15.102061 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b4fd2a7-ab0a-4fc0-9d42-d5f6ca1bdb62 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:15.102622 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b4fd2a7-ab0a-4fc0-9d42-d5f6ca1bdb62 admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:15.103895 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b4fd2a7-ab0a-4fc0-9d42-d5f6ca1bdb62 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:15.106860 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b4fd2a7-ab0a-4fc0-9d42-d5f6ca1bdb62 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:15.125148 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:15.132853 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1045/2063] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:57:14 2022] POST /identity/v3/users => generated 287 bytes in 138 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:57:15.136337 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d10267b4-b7c8-4d8d-9046-aaba21bd9098 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:15.154568 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d10267b4-b7c8-4d8d-9046-aaba21bd9098 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:15.160177 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-768ac64e-cc6c-4299-bc39-8425296c7280 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:15.160930 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-768ac64e-cc6c-4299-bc39-8425296c7280 admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:15.161344 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-768ac64e-cc6c-4299-bc39-8425296c7280 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:15.161690 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-768ac64e-cc6c-4299-bc39-8425296c7280 admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:15.162430 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-768ac64e-cc6c-4299-bc39-8425296c7280 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:15.164508 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-768ac64e-cc6c-4299-bc39-8425296c7280 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:15.178349 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1019/2064] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:57:15 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:57:15.180294 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b4fd2a7-ab0a-4fc0-9d42-d5f6ca1bdb62 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:15.184641 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-768ac64e-cc6c-4299-bc39-8425296c7280 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:15.192753 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-768ac64e-cc6c-4299-bc39-8425296c7280 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:15.193649 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b4fd2a7-ab0a-4fc0-9d42-d5f6ca1bdb62 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:15.200143 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-080dc79f-61e6-4318-a42f-3a004bfe9045 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:15.200876 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-080dc79f-61e6-4318-a42f-3a004bfe9045 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:15.201301 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-080dc79f-61e6-4318-a42f-3a004bfe9045 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:15.201671 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-080dc79f-61e6-4318-a42f-3a004bfe9045 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:15.202490 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-080dc79f-61e6-4318-a42f-3a004bfe9045 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:15.204480 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-080dc79f-61e6-4318-a42f-3a004bfe9045 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:15.207774 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-768ac64e-cc6c-4299-bc39-8425296c7280 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:15.215504 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cd99dae-d3a0-49e4-a7b6-9d50eeae91cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:15.216294 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cd99dae-d3a0-49e4-a7b6-9d50eeae91cf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:15.216575 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cd99dae-d3a0-49e4-a7b6-9d50eeae91cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:15.216855 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cd99dae-d3a0-49e4-a7b6-9d50eeae91cf admin admin] PATH_INFO: `/v3/projects/30f25c4e8f0046cfac8a8398ffa075e3` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:15.218171 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cd99dae-d3a0-49e4-a7b6-9d50eeae91cf admin admin] RBAC: Authorizing `identity:delete_project(project_id=30f25c4e8f0046cfac8a8398ffa075e3)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:15.220252 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cd99dae-d3a0-49e4-a7b6-9d50eeae91cf admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:15.222612 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1046/2065] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:57:15 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 09 00:57:15.224353 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-080dc79f-61e6-4318-a42f-3a004bfe9045 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:15.227645 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-080dc79f-61e6-4318-a42f-3a004bfe9045 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:15.242682 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-080dc79f-61e6-4318-a42f-3a004bfe9045 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:15.247993 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-143add76-27e9-4dd1-8a3d-259ba5310e55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:15.248647 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143add76-27e9-4dd1-8a3d-259ba5310e55 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:15.249060 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143add76-27e9-4dd1-8a3d-259ba5310e55 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:15.249337 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143add76-27e9-4dd1-8a3d-259ba5310e55 admin admin] PATH_INFO: `/v3/projects/619223fd7bd04d9e9d9f1a866bd1e2aa` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:15.250763 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-143add76-27e9-4dd1-8a3d-259ba5310e55 admin admin] RBAC: Authorizing `identity:delete_project(project_id=619223fd7bd04d9e9d9f1a866bd1e2aa)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:15.253792 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-143add76-27e9-4dd1-8a3d-259ba5310e55 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:15.273299 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-8cd99dae-d3a0-49e4-a7b6-9d50eeae91cf admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '30f25c4e8f0046cfac8a8398ffa075e3', 'request_id': 'req-8cd99dae-d3a0-49e4-a7b6-9d50eeae91cf'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:15.279507 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-8cd99dae-d3a0-49e4-a7b6-9d50eeae91cf admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '30f25c4e8f0046cfac8a8398ffa075e3', 'request_id': 'req-8cd99dae-d3a0-49e4-a7b6-9d50eeae91cf'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:15.288431 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-8cd99dae-d3a0-49e4-a7b6-9d50eeae91cf admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '30f25c4e8f0046cfac8a8398ffa075e3', 'request_id': 'req-8cd99dae-d3a0-49e4-a7b6-9d50eeae91cf'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:15.301655 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-8cd99dae-d3a0-49e4-a7b6-9d50eeae91cf admin admin] The token cache is being invalidate because project 30f25c4e8f0046cfac8a8398ffa075e3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81453) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 00:57:15.301946 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-8cd99dae-d3a0-49e4-a7b6-9d50eeae91cf admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:15.302596 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-8cd99dae-d3a0-49e4-a7b6-9d50eeae91cf admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:15.304194 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1020/2066] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 00:57:15 2022] DELETE /identity/v3/projects/30f25c4e8f0046cfac8a8398ffa075e3 => generated 0 bytes in 120 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:15.304909 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cd99dae-d3a0-49e4-a7b6-9d50eeae91cf admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:15.308319 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-8cd99dae-d3a0-49e4-a7b6-9d50eeae91cf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:15.309550 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-143add76-27e9-4dd1-8a3d-259ba5310e55 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '619223fd7bd04d9e9d9f1a866bd1e2aa', 'request_id': 'req-143add76-27e9-4dd1-8a3d-259ba5310e55'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:15.316111 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-143add76-27e9-4dd1-8a3d-259ba5310e55 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '619223fd7bd04d9e9d9f1a866bd1e2aa', 'request_id': 'req-143add76-27e9-4dd1-8a3d-259ba5310e55'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:15.323287 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-143add76-27e9-4dd1-8a3d-259ba5310e55 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '619223fd7bd04d9e9d9f1a866bd1e2aa', 'request_id': 'req-143add76-27e9-4dd1-8a3d-259ba5310e55'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:15.338647 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-143add76-27e9-4dd1-8a3d-259ba5310e55 admin admin] The token cache is being invalidate because project 619223fd7bd04d9e9d9f1a866bd1e2aa was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 00:57:15.338862 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-143add76-27e9-4dd1-8a3d-259ba5310e55 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:15.339708 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-143add76-27e9-4dd1-8a3d-259ba5310e55 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:15.340416 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:DELETE=10,keystone:SELECT=88,keystone:INSERT=2 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:15.344856 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1047/2067] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 00:57:15 2022] DELETE /identity/v3/projects/619223fd7bd04d9e9d9f1a866bd1e2aa => generated 0 bytes in 119 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:15.344856 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a384d70c-7510-4f93-b47c-3fa3f727d05a None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:15.344856 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a384d70c-7510-4f93-b47c-3fa3f727d05a None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:15.344856 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a384d70c-7510-4f93-b47c-3fa3f727d05a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:15.367427 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-a384d70c-7510-4f93-b47c-3fa3f727d05a None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:15.368685 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-a384d70c-7510-4f93-b47c-3fa3f727d05a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:15.396765 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cd99dae-d3a0-49e4-a7b6-9d50eeae91cf admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:15.473855 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1048/2068] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:57:15.474881 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a384d70c-7510-4f93-b47c-3fa3f727d05a None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:15.476910 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-a384d70c-7510-4f93-b47c-3fa3f727d05a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:15.485348 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a3ca3aa-e1fb-43d4-8e32-bfc0fd66876f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:15.486942 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a3ca3aa-e1fb-43d4-8e32-bfc0fd66876f admin admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:15.487610 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a3ca3aa-e1fb-43d4-8e32-bfc0fd66876f admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:15.488032 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a3ca3aa-e1fb-43d4-8e32-bfc0fd66876f admin admin] PATH_INFO: `/v3/projects/c15c3df3639a48dc9d1276290b1a0173/users/ee8c3cfe357c4d3183d3cba87646d289/roles/588016933bab436d8b7c3ac76827356d` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:15.490011 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a384d70c-7510-4f93-b47c-3fa3f727d05a None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:15.495355 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fe39147-e7be-41c6-a37a-ebdc1c1e044e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:15.496169 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe39147-e7be-41c6-a37a-ebdc1c1e044e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:15.496456 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe39147-e7be-41c6-a37a-ebdc1c1e044e admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:15.496732 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe39147-e7be-41c6-a37a-ebdc1c1e044e admin admin] PATH_INFO: `/v3/users/fdfcc4b4521743fcb7b4a804fe229322` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:15.498096 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fe39147-e7be-41c6-a37a-ebdc1c1e044e admin admin] RBAC: Authorizing `identity:delete_user(user_id=fdfcc4b4521743fcb7b4a804fe229322)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:15.500112 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fe39147-e7be-41c6-a37a-ebdc1c1e044e admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:15.510175 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a3ca3aa-e1fb-43d4-8e32-bfc0fd66876f admin admin] RBAC: Authorizing `identity:create_grant(project_id=c15c3df3639a48dc9d1276290b1a0173, user_id=ee8c3cfe357c4d3183d3cba87646d289, role_id=588016933bab436d8b7c3ac76827356d)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:15.512994 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a3ca3aa-e1fb-43d4-8e32-bfc0fd66876f admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:15.535636 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1021/2069] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:57:15 2022] PUT /identity/v3/projects/c15c3df3639a48dc9d1276290b1a0173/users/ee8c3cfe357c4d3183d3cba87646d289/roles/588016933bab436d8b7c3ac76827356d => generated 0 bytes in 231 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:15.537031 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e445d90e-e461-408c-8cf3-cf04dc6b35be None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:15.537318 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e445d90e-e461-408c-8cf3-cf04dc6b35be None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:15.537591 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e445d90e-e461-408c-8cf3-cf04dc6b35be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:15.565429 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-e445d90e-e461-408c-8cf3-cf04dc6b35be None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:15.566887 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-e445d90e-e461-408c-8cf3-cf04dc6b35be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:15.576002 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-3fe39147-e7be-41c6-a37a-ebdc1c1e044e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fdfcc4b4521743fcb7b4a804fe229322'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:15.585960 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-3fe39147-e7be-41c6-a37a-ebdc1c1e044e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fdfcc4b4521743fcb7b4a804fe229322'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:15.586742 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-3fe39147-e7be-41c6-a37a-ebdc1c1e044e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fdfcc4b4521743fcb7b4a804fe229322'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:15.587525 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-3fe39147-e7be-41c6-a37a-ebdc1c1e044e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fdfcc4b4521743fcb7b4a804fe229322'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:15.597256 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-3fe39147-e7be-41c6-a37a-ebdc1c1e044e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fdfcc4b4521743fcb7b4a804fe229322'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:15.602698 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=83,keystone:DELETE=15,keystone:INSERT=2 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:15.624845 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1049/2070] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 00:57:15 2022] DELETE /identity/v3/users/fdfcc4b4521743fcb7b4a804fe229322 => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:15.626772 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecfee180-d0d5-4f44-862b-c2faa4e47fe1 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:15.627089 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecfee180-d0d5-4f44-862b-c2faa4e47fe1 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:15.627362 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecfee180-d0d5-4f44-862b-c2faa4e47fe1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:15.656065 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1022/2071] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:57:15.662323 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e445d90e-e461-408c-8cf3-cf04dc6b35be None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:15.671098 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-ecfee180-d0d5-4f44-862b-c2faa4e47fe1 None None] MFA Rules not processed for user `ee8c3cfe357c4d3183d3cba87646d289`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:15.672492 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-ecfee180-d0d5-4f44-862b-c2faa4e47fe1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:15.675638 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e445d90e-e461-408c-8cf3-cf04dc6b35be None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:15.749507 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:15.758127 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1050/2072] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:15 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:57:15.760494 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1c20408-8226-44ea-9066-cb99d891608a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:15.761765 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c20408-8226-44ea-9066-cb99d891608a admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:15.762049 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c20408-8226-44ea-9066-cb99d891608a admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:15.762325 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c20408-8226-44ea-9066-cb99d891608a admin admin] PATH_INFO: `/v3/domains` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:15.763831 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecfee180-d0d5-4f44-862b-c2faa4e47fe1 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:15.764964 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1c20408-8226-44ea-9066-cb99d891608a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:57:15.765279 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1c20408-8226-44ea-9066-cb99d891608a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:15.766861 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1c20408-8226-44ea-9066-cb99d891608a admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:15.769880 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-ecfee180-d0d5-4f44-862b-c2faa4e47fe1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:15.783163 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1023/2073] 10.222.0.33 () {58 vars in 1153 bytes} [Sun Oct 9 00:57:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 122 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:57:15.788197 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1c20408-8226-44ea-9066-cb99d891608a admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:15.792233 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecfee180-d0d5-4f44-862b-c2faa4e47fe1 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:15.798047 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee95fb47-08fd-4dcf-8df3-8c3cb35709b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:15.798806 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee95fb47-08fd-4dcf-8df3-8c3cb35709b4 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:15.799071 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee95fb47-08fd-4dcf-8df3-8c3cb35709b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:15.799259 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee95fb47-08fd-4dcf-8df3-8c3cb35709b4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:15.800043 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee95fb47-08fd-4dcf-8df3-8c3cb35709b4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:15.801357 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1c20408-8226-44ea-9066-cb99d891608a admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:15.802322 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee95fb47-08fd-4dcf-8df3-8c3cb35709b4 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:15.806670 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7c7a72d-330f-4c29-bab7-51ea05e73bb0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:15.807297 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7c7a72d-330f-4c29-bab7-51ea05e73bb0 admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:15.807581 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7c7a72d-330f-4c29-bab7-51ea05e73bb0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:15.807886 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7c7a72d-330f-4c29-bab7-51ea05e73bb0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:15.808525 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7c7a72d-330f-4c29-bab7-51ea05e73bb0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:15.810882 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7c7a72d-330f-4c29-bab7-51ea05e73bb0 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:15.843569 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1051/2074] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 00:57:15 2022] POST /identity/v3/projects => generated 371 bytes in 80 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:57:15.847644 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee95fb47-08fd-4dcf-8df3-8c3cb35709b4 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:15.849300 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1024/2075] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 00:57:15 2022] POST /identity/v3/projects => generated 385 bytes in 61 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:57:15.850249 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7c7a72d-330f-4c29-bab7-51ea05e73bb0 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:15.925994 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7c7a72d-330f-4c29-bab7-51ea05e73bb0 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:15.930979 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34ec09b2-7aa5-4b08-bb31-4ca7ac8c2e23 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:15.932166 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34ec09b2-7aa5-4b08-bb31-4ca7ac8c2e23 admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:15.932306 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee95fb47-08fd-4dcf-8df3-8c3cb35709b4 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:15.932524 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34ec09b2-7aa5-4b08-bb31-4ca7ac8c2e23 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:15.932713 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34ec09b2-7aa5-4b08-bb31-4ca7ac8c2e23 admin admin] PATH_INFO: `/v3/users` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:15.933496 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34ec09b2-7aa5-4b08-bb31-4ca7ac8c2e23 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:15.935904 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34ec09b2-7aa5-4b08-bb31-4ca7ac8c2e23 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:15.937560 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cf12fd1-289a-4c60-93ac-0a3b3ab77cf7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:15.938166 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cf12fd1-289a-4c60-93ac-0a3b3ab77cf7 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:15.938659 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cf12fd1-289a-4c60-93ac-0a3b3ab77cf7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:15.938962 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cf12fd1-289a-4c60-93ac-0a3b3ab77cf7 admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:15.939631 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cf12fd1-289a-4c60-93ac-0a3b3ab77cf7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:15.942067 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cf12fd1-289a-4c60-93ac-0a3b3ab77cf7 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:15.964485 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1025/2076] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:57:15 2022] POST /identity/v3/users => generated 293 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:57:15.965536 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34ec09b2-7aa5-4b08-bb31-4ca7ac8c2e23 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:15.968650 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-34ec09b2-7aa5-4b08-bb31-4ca7ac8c2e23 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:15.979106 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1052/2077] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:57:15 2022] POST /identity/v3/users => generated 286 bytes in 132 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:57:15.980045 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cf12fd1-289a-4c60-93ac-0a3b3ab77cf7 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:15.983730 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34ec09b2-7aa5-4b08-bb31-4ca7ac8c2e23 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:15.990443 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9ba40c8-4f68-4a54-b4a1-5c1142e9efa2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:15.991218 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ba40c8-4f68-4a54-b4a1-5c1142e9efa2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:15.991338 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cf12fd1-289a-4c60-93ac-0a3b3ab77cf7 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:15.991432 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ba40c8-4f68-4a54-b4a1-5c1142e9efa2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:15.991650 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ba40c8-4f68-4a54-b4a1-5c1142e9efa2 admin admin] PATH_INFO: `/v3/projects/dcc60c20223e4b8892d82f207177f9cc` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:15.992760 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9ba40c8-4f68-4a54-b4a1-5c1142e9efa2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=dcc60c20223e4b8892d82f207177f9cc)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:15.994655 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9ba40c8-4f68-4a54-b4a1-5c1142e9efa2 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:16.001035 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0721cae1-5dac-4036-9d6b-204ed2d299f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:16.001851 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0721cae1-5dac-4036-9d6b-204ed2d299f5 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:16.002139 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0721cae1-5dac-4036-9d6b-204ed2d299f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:16.002446 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0721cae1-5dac-4036-9d6b-204ed2d299f5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:16.003912 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0721cae1-5dac-4036-9d6b-204ed2d299f5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:16.016250 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0721cae1-5dac-4036-9d6b-204ed2d299f5 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:16.031619 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1053/2078] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:57:15 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:57:16.031619 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0721cae1-5dac-4036-9d6b-204ed2d299f5 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:16.043523 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0721cae1-5dac-4036-9d6b-204ed2d299f5 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:16.057735 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-d9ba40c8-4f68-4a54-b4a1-5c1142e9efa2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'dcc60c20223e4b8892d82f207177f9cc', 'request_id': 'req-d9ba40c8-4f68-4a54-b4a1-5c1142e9efa2'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:16.062941 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-d9ba40c8-4f68-4a54-b4a1-5c1142e9efa2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'dcc60c20223e4b8892d82f207177f9cc', 'request_id': 'req-d9ba40c8-4f68-4a54-b4a1-5c1142e9efa2'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:16.070586 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-d9ba40c8-4f68-4a54-b4a1-5c1142e9efa2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'dcc60c20223e4b8892d82f207177f9cc', 'request_id': 'req-d9ba40c8-4f68-4a54-b4a1-5c1142e9efa2'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:16.084330 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:DELETE=2,keystone:SELECT=94,keystone:INSERT=4 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:16.089796 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-d9ba40c8-4f68-4a54-b4a1-5c1142e9efa2 admin admin] The token cache is being invalidate because project dcc60c20223e4b8892d82f207177f9cc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81453) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 00:57:16.090122 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-d9ba40c8-4f68-4a54-b4a1-5c1142e9efa2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:16.090813 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-d9ba40c8-4f68-4a54-b4a1-5c1142e9efa2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:16.095505 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1026/2079] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 00:57:15 2022] DELETE /identity/v3/projects/dcc60c20223e4b8892d82f207177f9cc => generated 0 bytes in 127 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:16.096342 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9ba40c8-4f68-4a54-b4a1-5c1142e9efa2 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:16.098296 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-d9ba40c8-4f68-4a54-b4a1-5c1142e9efa2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:16.122714 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fdbff3a-e471-4418-a720-18f58a98a662 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:16.123644 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fdbff3a-e471-4418-a720-18f58a98a662 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:16.123644 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fdbff3a-e471-4418-a720-18f58a98a662 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:16.123869 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fdbff3a-e471-4418-a720-18f58a98a662 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:16.124503 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fdbff3a-e471-4418-a720-18f58a98a662 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:16.126118 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fdbff3a-e471-4418-a720-18f58a98a662 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:16.145548 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1054/2080] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:57:16 2022] GET /identity/v3/roles => generated 1420 bytes in 113 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:57:16.145548 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fdbff3a-e471-4418-a720-18f58a98a662 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:16.147288 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-3fdbff3a-e471-4418-a720-18f58a98a662 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:16.159654 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=88,keystone:INSERT=5,keystone:DELETE=7 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:16.160496 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fdbff3a-e471-4418-a720-18f58a98a662 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:16.167120 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ea1c877-d520-4624-8ab4-47f90f1cbb5e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:16.168008 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea1c877-d520-4624-8ab4-47f90f1cbb5e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:16.168341 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea1c877-d520-4624-8ab4-47f90f1cbb5e admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:16.168615 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea1c877-d520-4624-8ab4-47f90f1cbb5e admin admin] PATH_INFO: `/v3/users/5b2c6b9740a14ee4bfeac6e3c45eec2f` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:16.170290 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ea1c877-d520-4624-8ab4-47f90f1cbb5e admin admin] RBAC: Authorizing `identity:delete_user(user_id=5b2c6b9740a14ee4bfeac6e3c45eec2f)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:16.172618 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ea1c877-d520-4624-8ab4-47f90f1cbb5e admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:16.182539 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9ba40c8-4f68-4a54-b4a1-5c1142e9efa2 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:16.187557 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fe7294c-fbef-4dec-b16d-ee698f42c514 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:16.188188 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe7294c-fbef-4dec-b16d-ee698f42c514 admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:16.188467 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe7294c-fbef-4dec-b16d-ee698f42c514 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:16.188744 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe7294c-fbef-4dec-b16d-ee698f42c514 admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:16.189571 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fe7294c-fbef-4dec-b16d-ee698f42c514 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:16.191751 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fe7294c-fbef-4dec-b16d-ee698f42c514 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:16.202875 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1027/2081] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:57:16 2022] GET /identity/v3/roles => generated 1420 bytes in 107 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:57:16.204627 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fe7294c-fbef-4dec-b16d-ee698f42c514 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:16.206576 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-4fe7294c-fbef-4dec-b16d-ee698f42c514 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:16.221581 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fe7294c-fbef-4dec-b16d-ee698f42c514 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:16.226679 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91f42d0f-09cb-41ed-b904-24a27b9f6579 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:16.226842 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-9ea1c877-d520-4624-8ab4-47f90f1cbb5e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5b2c6b9740a14ee4bfeac6e3c45eec2f'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:16.227284 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91f42d0f-09cb-41ed-b904-24a27b9f6579 admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:16.227726 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91f42d0f-09cb-41ed-b904-24a27b9f6579 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:16.228119 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91f42d0f-09cb-41ed-b904-24a27b9f6579 admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:16.228755 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91f42d0f-09cb-41ed-b904-24a27b9f6579 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:16.230770 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91f42d0f-09cb-41ed-b904-24a27b9f6579 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:16.235964 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-9ea1c877-d520-4624-8ab4-47f90f1cbb5e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5b2c6b9740a14ee4bfeac6e3c45eec2f'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:16.236856 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-9ea1c877-d520-4624-8ab4-47f90f1cbb5e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5b2c6b9740a14ee4bfeac6e3c45eec2f'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:16.238418 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-9ea1c877-d520-4624-8ab4-47f90f1cbb5e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5b2c6b9740a14ee4bfeac6e3c45eec2f'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:16.245290 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1028/2082] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:57:16 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:57:16.245957 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91f42d0f-09cb-41ed-b904-24a27b9f6579 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:16.249640 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-91f42d0f-09cb-41ed-b904-24a27b9f6579 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:16.251076 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-9ea1c877-d520-4624-8ab4-47f90f1cbb5e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5b2c6b9740a14ee4bfeac6e3c45eec2f'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:16.262954 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91f42d0f-09cb-41ed-b904-24a27b9f6579 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:16.269073 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f409c342-5cb5-4d6b-b69a-38b031ac95ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:16.269884 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f409c342-5cb5-4d6b-b69a-38b031ac95ce admin admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:16.270260 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f409c342-5cb5-4d6b-b69a-38b031ac95ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:16.270536 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f409c342-5cb5-4d6b-b69a-38b031ac95ce admin admin] PATH_INFO: `/v3/projects/91d97c3bee2d48ecac465ead2ab9b24a/users/73bdf78202504dbe8356b731c7c1c57b/roles/588016933bab436d8b7c3ac76827356d` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:16.277016 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1055/2083] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 00:57:16 2022] DELETE /identity/v3/users/5b2c6b9740a14ee4bfeac6e3c45eec2f => generated 0 bytes in 134 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:16.284540 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ea1c877-d520-4624-8ab4-47f90f1cbb5e admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:16.287532 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-9ea1c877-d520-4624-8ab4-47f90f1cbb5e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:16.291111 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f409c342-5cb5-4d6b-b69a-38b031ac95ce admin admin] RBAC: Authorizing `identity:create_grant(project_id=91d97c3bee2d48ecac465ead2ab9b24a, user_id=73bdf78202504dbe8356b731c7c1c57b, role_id=588016933bab436d8b7c3ac76827356d)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:16.293984 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f409c342-5cb5-4d6b-b69a-38b031ac95ce admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:16.318947 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1029/2084] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:57:16 2022] PUT /identity/v3/projects/91d97c3bee2d48ecac465ead2ab9b24a/users/73bdf78202504dbe8356b731c7c1c57b/roles/588016933bab436d8b7c3ac76827356d => generated 0 bytes in 64 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:16.320951 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb357be-24e1-4b32-b51c-c750883cfb27 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:16.321466 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb357be-24e1-4b32-b51c-c750883cfb27 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:16.321996 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb357be-24e1-4b32-b51c-c750883cfb27 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:16.352836 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-1bb357be-24e1-4b32-b51c-c750883cfb27 None None] MFA Rules not processed for user `73bdf78202504dbe8356b731c7c1c57b`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:16.356191 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-1bb357be-24e1-4b32-b51c-c750883cfb27 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:16.382262 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ea1c877-d520-4624-8ab4-47f90f1cbb5e admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:16.439656 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1030/2085] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:16 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 09 00:57:16.449334 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bb357be-24e1-4b32-b51c-c750883cfb27 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:16.450755 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:16.453055 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-1bb357be-24e1-4b32-b51c-c750883cfb27 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:16.474632 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d6c2cf9-b76e-429b-b749-ae7baa169a57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:16.475515 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d6c2cf9-b76e-429b-b749-ae7baa169a57 admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:16.476126 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d6c2cf9-b76e-429b-b749-ae7baa169a57 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:16.476469 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d6c2cf9-b76e-429b-b749-ae7baa169a57 admin admin] PATH_INFO: `/v3/projects/b0adffcc6be142ee987cf24ca2c646b1/users/65375e71d0d940658ef71f9f3f613ec0/roles/588016933bab436d8b7c3ac76827356d` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:16.507239 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d6c2cf9-b76e-429b-b749-ae7baa169a57 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b0adffcc6be142ee987cf24ca2c646b1, user_id=65375e71d0d940658ef71f9f3f613ec0, role_id=588016933bab436d8b7c3ac76827356d)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:16.507239 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d6c2cf9-b76e-429b-b749-ae7baa169a57 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:16.516908 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1056/2086] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:57:16 2022] PUT /identity/v3/projects/b0adffcc6be142ee987cf24ca2c646b1/users/65375e71d0d940658ef71f9f3f613ec0/roles/588016933bab436d8b7c3ac76827356d => generated 0 bytes in 239 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:16.523413 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bb357be-24e1-4b32-b51c-c750883cfb27 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:16.527372 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-755ee72f-dda5-419c-8dae-9300ff0d24b0 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:16.527833 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-755ee72f-dda5-419c-8dae-9300ff0d24b0 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:16.528341 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-755ee72f-dda5-419c-8dae-9300ff0d24b0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:16.559876 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-755ee72f-dda5-419c-8dae-9300ff0d24b0 None None] MFA Rules not processed for user `65375e71d0d940658ef71f9f3f613ec0`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:16.562174 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-755ee72f-dda5-419c-8dae-9300ff0d24b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:16.577704 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:16.583960 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9adf6999-fe7d-4ae2-9f58-1a88cd61ded5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3f48335668a4e8f934ac11ff66cd308', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:16.584206 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9adf6999-fe7d-4ae2-9f58-1a88cd61ded5 service cinder] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:16.584464 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9adf6999-fe7d-4ae2-9f58-1a88cd61ded5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:16.584734 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9adf6999-fe7d-4ae2-9f58-1a88cd61ded5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:16.655051 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1057/2087] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:16 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 09 00:57:16.657393 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-755ee72f-dda5-419c-8dae-9300ff0d24b0 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:16.664374 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-755ee72f-dda5-419c-8dae-9300ff0d24b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:16.667494 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9adf6999-fe7d-4ae2-9f58-1a88cd61ded5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:16.669386 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9adf6999-fe7d-4ae2-9f58-1a88cd61ded5 service cinder] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:16.683885 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1031/2088] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:57:16 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 235 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 09 00:57:16.684966 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9adf6999-fe7d-4ae2-9f58-1a88cd61ded5 service cinder] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:16.766215 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-755ee72f-dda5-419c-8dae-9300ff0d24b0 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:16.774264 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05f49b43-798a-4b30-9667-3e5b947b994c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:16.775151 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f49b43-798a-4b30-9667-3e5b947b994c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:16.775639 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f49b43-798a-4b30-9667-3e5b947b994c admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:16.776068 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f49b43-798a-4b30-9667-3e5b947b994c admin admin] PATH_INFO: `/v3/projects/aad4ea28111941a19ea89bf076cc9100` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:16.777909 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05f49b43-798a-4b30-9667-3e5b947b994c admin admin] RBAC: Authorizing `identity:delete_project(project_id=aad4ea28111941a19ea89bf076cc9100)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:16.780041 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05f49b43-798a-4b30-9667-3e5b947b994c admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:16.786758 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9adf6999-fe7d-4ae2-9f58-1a88cd61ded5 service cinder] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:16.795101 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06457bcf-cc1b-4b5e-9195-7e97b319aa0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:16.795875 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06457bcf-cc1b-4b5e-9195-7e97b319aa0f admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:16.796337 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06457bcf-cc1b-4b5e-9195-7e97b319aa0f admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:16.796507 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06457bcf-cc1b-4b5e-9195-7e97b319aa0f admin admin] PATH_INFO: `/v3/projects` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:16.797263 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06457bcf-cc1b-4b5e-9195-7e97b319aa0f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:16.804922 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06457bcf-cc1b-4b5e-9195-7e97b319aa0f admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:16.842971 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1032/2089] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 00:57:16 2022] POST /identity/v3/projects => generated 373 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:57:16.848642 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06457bcf-cc1b-4b5e-9195-7e97b319aa0f admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:16.854060 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-05f49b43-798a-4b30-9667-3e5b947b994c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'aad4ea28111941a19ea89bf076cc9100', 'request_id': 'req-05f49b43-798a-4b30-9667-3e5b947b994c'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:16.861325 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-05f49b43-798a-4b30-9667-3e5b947b994c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'aad4ea28111941a19ea89bf076cc9100', 'request_id': 'req-05f49b43-798a-4b30-9667-3e5b947b994c'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:16.864391 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=93,keystone:INSERT=1,keystone:DELETE=6 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:16.870362 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-05f49b43-798a-4b30-9667-3e5b947b994c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'aad4ea28111941a19ea89bf076cc9100', 'request_id': 'req-05f49b43-798a-4b30-9667-3e5b947b994c'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:16.885753 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-05f49b43-798a-4b30-9667-3e5b947b994c admin admin] The token cache is being invalidate because project aad4ea28111941a19ea89bf076cc9100 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 00:57:16.886047 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-05f49b43-798a-4b30-9667-3e5b947b994c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:16.886690 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-05f49b43-798a-4b30-9667-3e5b947b994c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:16.892664 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1058/2090] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 00:57:16 2022] DELETE /identity/v3/projects/aad4ea28111941a19ea89bf076cc9100 => generated 0 bytes in 232 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:16.894836 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f1d800-5506-44f1-b9ee-6a4eee894795 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:16.895247 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f1d800-5506-44f1-b9ee-6a4eee894795 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:16.895490 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f1d800-5506-44f1-b9ee-6a4eee894795 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:16.920016 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-b1f1d800-5506-44f1-b9ee-6a4eee894795 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:16.921698 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-b1f1d800-5506-44f1-b9ee-6a4eee894795 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:16.953873 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06457bcf-cc1b-4b5e-9195-7e97b319aa0f admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:16.961076 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46a104af-9070-4484-9ca0-6abf424ad391 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:16.962049 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a104af-9070-4484-9ca0-6abf424ad391 admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:16.962564 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a104af-9070-4484-9ca0-6abf424ad391 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:16.963020 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a104af-9070-4484-9ca0-6abf424ad391 admin admin] PATH_INFO: `/v3/users` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:16.963917 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46a104af-9070-4484-9ca0-6abf424ad391 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:16.966047 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46a104af-9070-4484-9ca0-6abf424ad391 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:16.999785 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1033/2091] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:57:16 2022] POST /identity/v3/users => generated 293 bytes in 152 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:57:17.005125 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1059/2092] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:57:17.005703 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46a104af-9070-4484-9ca0-6abf424ad391 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:17.007900 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-46a104af-9070-4484-9ca0-6abf424ad391 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:17.013261 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0422a05f-de95-4c66-a286-e6650f220d19 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:17.013620 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0422a05f-de95-4c66-a286-e6650f220d19 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:17.013970 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0422a05f-de95-4c66-a286-e6650f220d19 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:17.027197 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46a104af-9070-4484-9ca0-6abf424ad391 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:17.033779 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1907911d-c2c9-436b-8a00-d9993c1fcc5a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:17.034433 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1907911d-c2c9-436b-8a00-d9993c1fcc5a admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:17.035531 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1907911d-c2c9-436b-8a00-d9993c1fcc5a admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:17.035848 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1907911d-c2c9-436b-8a00-d9993c1fcc5a admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:17.036643 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1907911d-c2c9-436b-8a00-d9993c1fcc5a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:17.038202 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1907911d-c2c9-436b-8a00-d9993c1fcc5a admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:17.039320 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-0422a05f-de95-4c66-a286-e6650f220d19 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:17.041176 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-0422a05f-de95-4c66-a286-e6650f220d19 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:17.053890 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1034/2093] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:57:17 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:57:17.059827 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1060/2094] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 47 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 09 00:57:17.066179 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0422a05f-de95-4c66-a286-e6650f220d19 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:17.067487 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1907911d-c2c9-436b-8a00-d9993c1fcc5a admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:17.081070 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0422a05f-de95-4c66-a286-e6650f220d19 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:17.088383 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80aea030-7feb-444a-9fc8-ed6fed954875 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:17.089302 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80aea030-7feb-444a-9fc8-ed6fed954875 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:17.089467 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80aea030-7feb-444a-9fc8-ed6fed954875 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:17.089740 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80aea030-7feb-444a-9fc8-ed6fed954875 admin admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:17.090476 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80aea030-7feb-444a-9fc8-ed6fed954875 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:57:17.090775 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80aea030-7feb-444a-9fc8-ed6fed954875 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:17.092690 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80aea030-7feb-444a-9fc8-ed6fed954875 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:17.098340 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1907911d-c2c9-436b-8a00-d9993c1fcc5a admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:17.107178 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1061/2095] 10.222.0.33 () {58 vars in 1153 bytes} [Sun Oct 9 00:57:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:57:17.113473 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80aea030-7feb-444a-9fc8-ed6fed954875 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:17.114787 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7592170e-58bf-4b60-adad-2a7e1c7b7a06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:17.116226 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7592170e-58bf-4b60-adad-2a7e1c7b7a06 admin admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:17.116507 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7592170e-58bf-4b60-adad-2a7e1c7b7a06 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:17.116778 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7592170e-58bf-4b60-adad-2a7e1c7b7a06 admin admin] PATH_INFO: `/v3/projects/f5a5a1e0aa9d43fba6e79cb7d035241f/users/10570712215149159e5517b255ec6637/roles/092036ee46ac427baf4d56468575f697` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:17.119308 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:17.130551 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80aea030-7feb-444a-9fc8-ed6fed954875 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:17.135681 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7592170e-58bf-4b60-adad-2a7e1c7b7a06 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f5a5a1e0aa9d43fba6e79cb7d035241f, user_id=10570712215149159e5517b255ec6637, role_id=092036ee46ac427baf4d56468575f697)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:17.137375 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee09912d-5651-4a25-8e8d-e7582ced3bcd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:17.138187 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee09912d-5651-4a25-8e8d-e7582ced3bcd admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:17.138580 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7592170e-58bf-4b60-adad-2a7e1c7b7a06 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:17.139010 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee09912d-5651-4a25-8e8d-e7582ced3bcd admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:17.139773 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee09912d-5651-4a25-8e8d-e7582ced3bcd admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:17.140601 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee09912d-5651-4a25-8e8d-e7582ced3bcd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:17.143761 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee09912d-5651-4a25-8e8d-e7582ced3bcd admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:17.160583 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1035/2096] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:57:17 2022] PUT /identity/v3/projects/f5a5a1e0aa9d43fba6e79cb7d035241f/users/10570712215149159e5517b255ec6637/roles/092036ee46ac427baf4d56468575f697 => generated 0 bytes in 96 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:17.164441 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7592170e-58bf-4b60-adad-2a7e1c7b7a06 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:17.177112 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1062/2097] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 00:57:17 2022] POST /identity/v3/projects => generated 373 bytes in 65 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 09 00:57:17.181097 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee09912d-5651-4a25-8e8d-e7582ced3bcd admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:17.228375 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7592170e-58bf-4b60-adad-2a7e1c7b7a06 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:17.244193 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee09912d-5651-4a25-8e8d-e7582ced3bcd admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:17.248882 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0803103-8681-4f86-91d5-eb6a0b32779e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:17.249477 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0803103-8681-4f86-91d5-eb6a0b32779e admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:17.249752 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0803103-8681-4f86-91d5-eb6a0b32779e admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:17.250021 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0803103-8681-4f86-91d5-eb6a0b32779e admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:17.250638 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0803103-8681-4f86-91d5-eb6a0b32779e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:17.252575 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0803103-8681-4f86-91d5-eb6a0b32779e admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:17.274069 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1063/2098] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:57:17 2022] POST /identity/v3/users => generated 287 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 09 00:57:17.278183 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0803103-8681-4f86-91d5-eb6a0b32779e admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:17.287225 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48dea053-8040-41c6-b2cf-72d9d405f817 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:17.287869 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48dea053-8040-41c6-b2cf-72d9d405f817 admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:17.287988 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0803103-8681-4f86-91d5-eb6a0b32779e admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:17.288149 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48dea053-8040-41c6-b2cf-72d9d405f817 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:17.288462 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48dea053-8040-41c6-b2cf-72d9d405f817 admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:17.289229 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48dea053-8040-41c6-b2cf-72d9d405f817 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:17.290815 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48dea053-8040-41c6-b2cf-72d9d405f817 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:17.292502 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e54eef2-306e-4faf-96fe-9eb3e5b077a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:17.293080 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e54eef2-306e-4faf-96fe-9eb3e5b077a7 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:17.293359 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e54eef2-306e-4faf-96fe-9eb3e5b077a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:17.293629 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e54eef2-306e-4faf-96fe-9eb3e5b077a7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:17.294255 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e54eef2-306e-4faf-96fe-9eb3e5b077a7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:17.295853 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e54eef2-306e-4faf-96fe-9eb3e5b077a7 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:17.301505 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1036/2099] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:57:17 2022] GET /identity/v3/roles => generated 1420 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 09 00:57:17.305376 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48dea053-8040-41c6-b2cf-72d9d405f817 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:17.309686 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1064/2100] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:57:17 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 09 00:57:17.310606 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e54eef2-306e-4faf-96fe-9eb3e5b077a7 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:17.315853 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48dea053-8040-41c6-b2cf-72d9d405f817 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:17.320426 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b2667a9-c71b-4b4b-ba11-8460de4c2d25 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:17.320796 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e54eef2-306e-4faf-96fe-9eb3e5b077a7 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:17.321061 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b2667a9-c71b-4b4b-ba11-8460de4c2d25 admin admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:17.321343 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b2667a9-c71b-4b4b-ba11-8460de4c2d25 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:17.321620 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b2667a9-c71b-4b4b-ba11-8460de4c2d25 admin admin] PATH_INFO: `/v3/projects/f5a5a1e0aa9d43fba6e79cb7d035241f/users/10570712215149159e5517b255ec6637/roles/092036ee46ac427baf4d56468575f697` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:17.323471 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b2667a9-c71b-4b4b-ba11-8460de4c2d25 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f5a5a1e0aa9d43fba6e79cb7d035241f, user_id=10570712215149159e5517b255ec6637, role_id=092036ee46ac427baf4d56468575f697)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:17.325320 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a324d80-e88e-4341-b74a-a149b874a4c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:17.325881 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a324d80-e88e-4341-b74a-a149b874a4c6 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:17.326129 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b2667a9-c71b-4b4b-ba11-8460de4c2d25 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:17.326224 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a324d80-e88e-4341-b74a-a149b874a4c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:17.326412 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a324d80-e88e-4341-b74a-a149b874a4c6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:17.327030 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a324d80-e88e-4341-b74a-a149b874a4c6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:17.329384 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a324d80-e88e-4341-b74a-a149b874a4c6 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:17.336340 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1037/2101] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:57:17 2022] PUT /identity/v3/projects/f5a5a1e0aa9d43fba6e79cb7d035241f/users/10570712215149159e5517b255ec6637/roles/092036ee46ac427baf4d56468575f697 => generated 0 bytes in 31 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 09 00:57:17.340186 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1065/2102] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:57:17 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 09 00:57:17.344175 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b2667a9-c71b-4b4b-ba11-8460de4c2d25 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:17.347766 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba831cf7-93dc-412f-ae8a-2e9a1ea4ae01 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:17.348055 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba831cf7-93dc-412f-ae8a-2e9a1ea4ae01 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:17.348326 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba831cf7-93dc-412f-ae8a-2e9a1ea4ae01 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:17.375898 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-ba831cf7-93dc-412f-ae8a-2e9a1ea4ae01 None None] MFA Rules not processed for user `10570712215149159e5517b255ec6637`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:17.377090 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-ba831cf7-93dc-412f-ae8a-2e9a1ea4ae01 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:17.381289 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=94,keystone:INSERT=5,keystone:DELETE=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:17.409688 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b2667a9-c71b-4b4b-ba11-8460de4c2d25 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:17.414819 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0935750-f252-4977-92f7-bf9196baa6bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:17.415476 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0935750-f252-4977-92f7-bf9196baa6bf admin admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:17.415786 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0935750-f252-4977-92f7-bf9196baa6bf admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:17.416063 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0935750-f252-4977-92f7-bf9196baa6bf admin admin] PATH_INFO: `/v3/projects/b827c14124484ee58548c6f686a007f8/users/283fde3ec93a4fe1abfaffc416eafe6e/roles/588016933bab436d8b7c3ac76827356d` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:17.430541 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0935750-f252-4977-92f7-bf9196baa6bf admin admin] RBAC: Authorizing `identity:create_grant(project_id=b827c14124484ee58548c6f686a007f8, user_id=283fde3ec93a4fe1abfaffc416eafe6e, role_id=588016933bab436d8b7c3ac76827356d)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:17.433299 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0935750-f252-4977-92f7-bf9196baa6bf admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:17.436417 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:17.449022 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1038/2103] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:57:17 2022] PUT /identity/v3/projects/b827c14124484ee58548c6f686a007f8/users/283fde3ec93a4fe1abfaffc416eafe6e/roles/588016933bab436d8b7c3ac76827356d => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:17.450655 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1066/2104] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:17 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 09 00:57:17.461093 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5cc9bf7-99c4-4e3a-867c-1c3342645e45 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:17.461459 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5cc9bf7-99c4-4e3a-867c-1c3342645e45 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:17.461730 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5cc9bf7-99c4-4e3a-867c-1c3342645e45 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:17.462377 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba831cf7-93dc-412f-ae8a-2e9a1ea4ae01 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:17.463861 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-ba831cf7-93dc-412f-ae8a-2e9a1ea4ae01 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:17.490441 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-a5cc9bf7-99c4-4e3a-867c-1c3342645e45 None None] MFA Rules not processed for user `283fde3ec93a4fe1abfaffc416eafe6e`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:17.491662 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-a5cc9bf7-99c4-4e3a-867c-1c3342645e45 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:17.529607 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba831cf7-93dc-412f-ae8a-2e9a1ea4ae01 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:17.575528 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1039/2105] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:17 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:57:17.587146 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3c7a3ec-bfe9-4b79-8f56-232e457899e3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3f48335668a4e8f934ac11ff66cd308', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:17.587820 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3c7a3ec-bfe9-4b79-8f56-232e457899e3 service cinder] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:17.588106 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3c7a3ec-bfe9-4b79-8f56-232e457899e3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:17.588492 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3c7a3ec-bfe9-4b79-8f56-232e457899e3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:17.590220 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-e3c7a3ec-bfe9-4b79-8f56-232e457899e3 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:17.601597 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5cc9bf7-99c4-4e3a-867c-1c3342645e45 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:17.602979 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-a5cc9bf7-99c4-4e3a-867c-1c3342645e45 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:17.615148 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5cc9bf7-99c4-4e3a-867c-1c3342645e45 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:17.618979 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-360d0b52-de56-454a-9fee-bb2aa644380d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3f48335668a4e8f934ac11ff66cd308', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:17.619580 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-360d0b52-de56-454a-9fee-bb2aa644380d service cinder] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:17.619946 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-360d0b52-de56-454a-9fee-bb2aa644380d service cinder] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:17.620220 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-360d0b52-de56-454a-9fee-bb2aa644380d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:17.630334 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-360d0b52-de56-454a-9fee-bb2aa644380d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:17.631800 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-360d0b52-de56-454a-9fee-bb2aa644380d service cinder] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:17.640812 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1040/2106] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:57:17 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 09 00:57:17.665459 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3c7a3ec-bfe9-4b79-8f56-232e457899e3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:17.665459 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3c7a3ec-bfe9-4b79-8f56-232e457899e3 service cinder] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:17.667133 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-360d0b52-de56-454a-9fee-bb2aa644380d service cinder] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:17.673516 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1067/2107] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:57:17 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 212 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 09 00:57:17.674371 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3c7a3ec-bfe9-4b79-8f56-232e457899e3 service cinder] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:17.676331 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-e3c7a3ec-bfe9-4b79-8f56-232e457899e3 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:17.691400 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:17.742473 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-360d0b52-de56-454a-9fee-bb2aa644380d service cinder] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:17.747403 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c63112d4-1455-4b8c-a480-79498fbdec69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:17.748085 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c63112d4-1455-4b8c-a480-79498fbdec69 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:17.748368 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c63112d4-1455-4b8c-a480-79498fbdec69 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:17.748642 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c63112d4-1455-4b8c-a480-79498fbdec69 admin admin] PATH_INFO: `/v3/users/283fde3ec93a4fe1abfaffc416eafe6e` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:17.749104 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3c7a3ec-bfe9-4b79-8f56-232e457899e3 service cinder] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:17.749776 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c63112d4-1455-4b8c-a480-79498fbdec69 admin admin] RBAC: Authorizing `identity:delete_user(user_id=283fde3ec93a4fe1abfaffc416eafe6e)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:17.751742 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c63112d4-1455-4b8c-a480-79498fbdec69 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:17.753944 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f87a3f99-f9a1-42d3-a7ed-9677515d4fd8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:17.754571 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f87a3f99-f9a1-42d3-a7ed-9677515d4fd8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:17.754850 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f87a3f99-f9a1-42d3-a7ed-9677515d4fd8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:17.755123 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f87a3f99-f9a1-42d3-a7ed-9677515d4fd8 admin admin] PATH_INFO: `/v3/users/cd80a4e0abf841c79e6b537410c5cb00` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:17.756276 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f87a3f99-f9a1-42d3-a7ed-9677515d4fd8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=cd80a4e0abf841c79e6b537410c5cb00)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:17.758181 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f87a3f99-f9a1-42d3-a7ed-9677515d4fd8 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:17.808526 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-c63112d4-1455-4b8c-a480-79498fbdec69 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '283fde3ec93a4fe1abfaffc416eafe6e'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:17.813390 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-f87a3f99-f9a1-42d3-a7ed-9677515d4fd8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cd80a4e0abf841c79e6b537410c5cb00'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:17.816094 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-c63112d4-1455-4b8c-a480-79498fbdec69 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '283fde3ec93a4fe1abfaffc416eafe6e'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:17.816935 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-c63112d4-1455-4b8c-a480-79498fbdec69 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '283fde3ec93a4fe1abfaffc416eafe6e'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:17.822837 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-f87a3f99-f9a1-42d3-a7ed-9677515d4fd8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cd80a4e0abf841c79e6b537410c5cb00'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:17.823645 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-f87a3f99-f9a1-42d3-a7ed-9677515d4fd8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cd80a4e0abf841c79e6b537410c5cb00'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:17.824272 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-f87a3f99-f9a1-42d3-a7ed-9677515d4fd8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cd80a4e0abf841c79e6b537410c5cb00'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:17.824913 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-c63112d4-1455-4b8c-a480-79498fbdec69 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '283fde3ec93a4fe1abfaffc416eafe6e'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:17.834450 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-f87a3f99-f9a1-42d3-a7ed-9677515d4fd8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cd80a4e0abf841c79e6b537410c5cb00'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:17.835542 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:17.842061 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-c63112d4-1455-4b8c-a480-79498fbdec69 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '283fde3ec93a4fe1abfaffc416eafe6e'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:17.851694 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1041/2108] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 00:57:17 2022] DELETE /identity/v3/users/283fde3ec93a4fe1abfaffc416eafe6e => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 09 00:57:17.856022 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c3304cd-5354-481b-9534-dfdc6ccc6d69 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:17.856632 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c3304cd-5354-481b-9534-dfdc6ccc6d69 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:17.857820 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c3304cd-5354-481b-9534-dfdc6ccc6d69 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:17.870227 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1068/2109] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 00:57:17 2022] DELETE /identity/v3/users/cd80a4e0abf841c79e6b537410c5cb00 => generated 0 bytes in 196 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:17.876615 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d9a2e45-bb1f-40d6-a1d1-368b9bdd2c17 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:17.876901 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d9a2e45-bb1f-40d6-a1d1-368b9bdd2c17 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:17.877175 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d9a2e45-bb1f-40d6-a1d1-368b9bdd2c17 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:17.885019 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-6c3304cd-5354-481b-9534-dfdc6ccc6d69 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:17.886535 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-6c3304cd-5354-481b-9534-dfdc6ccc6d69 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:17.899320 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-3d9a2e45-bb1f-40d6-a1d1-368b9bdd2c17 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:17.901111 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-3d9a2e45-bb1f-40d6-a1d1-368b9bdd2c17 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:17.962604 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1042/2110] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:57:17.974166 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c3304cd-5354-481b-9534-dfdc6ccc6d69 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:17.975650 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-6c3304cd-5354-481b-9534-dfdc6ccc6d69 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:18.015469 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1069/2111] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:57:18.025794 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d9a2e45-bb1f-40d6-a1d1-368b9bdd2c17 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:18.027279 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-3d9a2e45-bb1f-40d6-a1d1-368b9bdd2c17 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:18.060771 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c3304cd-5354-481b-9534-dfdc6ccc6d69 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:18.100549 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8b3350b-2f34-4e91-8aea-1d32c0a5ad61 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:18.101183 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8b3350b-2f34-4e91-8aea-1d32c0a5ad61 service neutron] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:18.101465 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8b3350b-2f34-4e91-8aea-1d32c0a5ad61 service neutron] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:18.101755 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8b3350b-2f34-4e91-8aea-1d32c0a5ad61 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:18.109652 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:18.126081 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8b3350b-2f34-4e91-8aea-1d32c0a5ad61 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:18.127544 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8b3350b-2f34-4e91-8aea-1d32c0a5ad61 service neutron] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:18.141893 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1043/2112] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:57:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 09 00:57:18.152722 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d9a2e45-bb1f-40d6-a1d1-368b9bdd2c17 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:18.159308 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0b529cf-81ff-4c9d-a007-706835ce64e7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:18.160145 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b529cf-81ff-4c9d-a007-706835ce64e7 service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:18.160426 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b529cf-81ff-4c9d-a007-706835ce64e7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:18.160695 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b529cf-81ff-4c9d-a007-706835ce64e7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:18.175809 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0b529cf-81ff-4c9d-a007-706835ce64e7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:18.177367 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0b529cf-81ff-4c9d-a007-706835ce64e7 service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:18.189308 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1070/2113] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:57:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:57:18.949951 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8b3350b-2f34-4e91-8aea-1d32c0a5ad61 service neutron] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:18.951437 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-b8b3350b-2f34-4e91-8aea-1d32c0a5ad61 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:18.965646 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0b529cf-81ff-4c9d-a007-706835ce64e7 service neutron] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:18.967225 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-f0b529cf-81ff-4c9d-a007-706835ce64e7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:18.982445 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8b3350b-2f34-4e91-8aea-1d32c0a5ad61 service neutron] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:18.984653 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0b529cf-81ff-4c9d-a007-706835ce64e7 service neutron] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:18.989352 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-161e7ee2-1852-4161-a83f-ed899138b5d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:18.990169 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-161e7ee2-1852-4161-a83f-ed899138b5d5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:18.990507 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-161e7ee2-1852-4161-a83f-ed899138b5d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:18.990844 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-161e7ee2-1852-4161-a83f-ed899138b5d5 admin admin] PATH_INFO: `/v3/projects/b827c14124484ee58548c6f686a007f8` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:18.992223 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-161e7ee2-1852-4161-a83f-ed899138b5d5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b827c14124484ee58548c6f686a007f8)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:18.992630 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0f78e76-cd31-461e-8941-3da15a1ba43d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:18.993271 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f78e76-cd31-461e-8941-3da15a1ba43d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:18.993545 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f78e76-cd31-461e-8941-3da15a1ba43d admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:18.993820 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f78e76-cd31-461e-8941-3da15a1ba43d admin admin] PATH_INFO: `/v3/projects/aaf669b496984e38b2d5621c650ab60d` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:18.994375 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-161e7ee2-1852-4161-a83f-ed899138b5d5 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:19.006827 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0f78e76-cd31-461e-8941-3da15a1ba43d admin admin] RBAC: Authorizing `identity:delete_project(project_id=aaf669b496984e38b2d5621c650ab60d)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:19.012174 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0f78e76-cd31-461e-8941-3da15a1ba43d admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:19.069555 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-161e7ee2-1852-4161-a83f-ed899138b5d5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b827c14124484ee58548c6f686a007f8', 'request_id': 'req-161e7ee2-1852-4161-a83f-ed899138b5d5'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:19.076691 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-161e7ee2-1852-4161-a83f-ed899138b5d5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b827c14124484ee58548c6f686a007f8', 'request_id': 'req-161e7ee2-1852-4161-a83f-ed899138b5d5'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:19.085865 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-161e7ee2-1852-4161-a83f-ed899138b5d5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b827c14124484ee58548c6f686a007f8', 'request_id': 'req-161e7ee2-1852-4161-a83f-ed899138b5d5'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:19.108048 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-161e7ee2-1852-4161-a83f-ed899138b5d5 admin admin] The token cache is being invalidate because project b827c14124484ee58548c6f686a007f8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81453) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 00:57:19.108344 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-161e7ee2-1852-4161-a83f-ed899138b5d5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:19.109094 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-161e7ee2-1852-4161-a83f-ed899138b5d5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:19.110964 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1044/2114] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 00:57:18 2022] DELETE /identity/v3/projects/b827c14124484ee58548c6f686a007f8 => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:19.116126 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c52790-37a3-47c2-8a87-bf4ffaf96db9 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:19.118295 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c52790-37a3-47c2-8a87-bf4ffaf96db9 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:19.118648 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c52790-37a3-47c2-8a87-bf4ffaf96db9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:19.144613 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-e0f78e76-cd31-461e-8941-3da15a1ba43d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'aaf669b496984e38b2d5621c650ab60d', 'request_id': 'req-e0f78e76-cd31-461e-8941-3da15a1ba43d'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:19.147578 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-20c52790-37a3-47c2-8a87-bf4ffaf96db9 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:19.148944 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-20c52790-37a3-47c2-8a87-bf4ffaf96db9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:19.164225 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:19.167308 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-e0f78e76-cd31-461e-8941-3da15a1ba43d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'aaf669b496984e38b2d5621c650ab60d', 'request_id': 'req-e0f78e76-cd31-461e-8941-3da15a1ba43d'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:19.177473 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-e0f78e76-cd31-461e-8941-3da15a1ba43d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'aaf669b496984e38b2d5621c650ab60d', 'request_id': 'req-e0f78e76-cd31-461e-8941-3da15a1ba43d'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:19.216441 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-e0f78e76-cd31-461e-8941-3da15a1ba43d admin admin] The token cache is being invalidate because project aaf669b496984e38b2d5621c650ab60d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 00:57:19.216816 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-e0f78e76-cd31-461e-8941-3da15a1ba43d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:19.217625 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-e0f78e76-cd31-461e-8941-3da15a1ba43d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:19.219838 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1071/2115] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 00:57:18 2022] DELETE /identity/v3/projects/aaf669b496984e38b2d5621c650ab60d => generated 0 bytes in 254 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 09 00:57:19.223558 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0f78e76-cd31-461e-8941-3da15a1ba43d admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:19.227453 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-e0f78e76-cd31-461e-8941-3da15a1ba43d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:19.287579 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1045/2116] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:57:19.295457 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8686b22-5793-4a62-b87d-1cfd1d58176f None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:19.295788 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8686b22-5793-4a62-b87d-1cfd1d58176f None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:19.296064 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8686b22-5793-4a62-b87d-1cfd1d58176f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:19.319060 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-e8686b22-5793-4a62-b87d-1cfd1d58176f None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:19.320340 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-e8686b22-5793-4a62-b87d-1cfd1d58176f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:19.330037 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1046/2117] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 09 00:57:19.334670 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8686b22-5793-4a62-b87d-1cfd1d58176f None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:19.339344 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0f78e76-cd31-461e-8941-3da15a1ba43d admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:19.351693 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8686b22-5793-4a62-b87d-1cfd1d58176f None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:19.356069 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17db52b4-d67b-4e7c-971a-e56f3b33fbd4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:19.358957 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17db52b4-d67b-4e7c-971a-e56f3b33fbd4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:19.359309 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17db52b4-d67b-4e7c-971a-e56f3b33fbd4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:19.359489 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94396204-1479-49f1-a5a4-5ea0f6f462ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:19.359717 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17db52b4-d67b-4e7c-971a-e56f3b33fbd4 admin admin] PATH_INFO: `/v3/users/114bcf4dd21e4f64980c9e41cfb4dab5` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:19.360067 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94396204-1479-49f1-a5a4-5ea0f6f462ff admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:19.360346 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94396204-1479-49f1-a5a4-5ea0f6f462ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:19.360617 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94396204-1479-49f1-a5a4-5ea0f6f462ff admin admin] PATH_INFO: `/v3/domains` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:19.361060 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17db52b4-d67b-4e7c-971a-e56f3b33fbd4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=114bcf4dd21e4f64980c9e41cfb4dab5)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:19.361287 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94396204-1479-49f1-a5a4-5ea0f6f462ff admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:57:19.361661 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94396204-1479-49f1-a5a4-5ea0f6f462ff admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:19.363750 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94396204-1479-49f1-a5a4-5ea0f6f462ff admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:19.364064 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17db52b4-d67b-4e7c-971a-e56f3b33fbd4 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:19.379660 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1047/2118] 10.222.0.33 () {58 vars in 1153 bytes} [Sun Oct 9 00:57:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:57:19.384407 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94396204-1479-49f1-a5a4-5ea0f6f462ff admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:19.396986 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94396204-1479-49f1-a5a4-5ea0f6f462ff admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:19.402995 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb89fb15-024c-404e-93ef-9c7ae7dccee4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:19.403611 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb89fb15-024c-404e-93ef-9c7ae7dccee4 admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:19.403931 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb89fb15-024c-404e-93ef-9c7ae7dccee4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:19.404206 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb89fb15-024c-404e-93ef-9c7ae7dccee4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:19.404868 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb89fb15-024c-404e-93ef-9c7ae7dccee4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:19.407667 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb89fb15-024c-404e-93ef-9c7ae7dccee4 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:19.426933 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-17db52b4-d67b-4e7c-971a-e56f3b33fbd4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '114bcf4dd21e4f64980c9e41cfb4dab5'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:19.438193 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1048/2119] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 00:57:19 2022] POST /identity/v3/projects => generated 385 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:57:19.439253 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb89fb15-024c-404e-93ef-9c7ae7dccee4 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:19.442494 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-fb89fb15-024c-404e-93ef-9c7ae7dccee4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:19.446306 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-17db52b4-d67b-4e7c-971a-e56f3b33fbd4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '114bcf4dd21e4f64980c9e41cfb4dab5'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:19.447120 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-17db52b4-d67b-4e7c-971a-e56f3b33fbd4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '114bcf4dd21e4f64980c9e41cfb4dab5'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:19.447829 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-17db52b4-d67b-4e7c-971a-e56f3b33fbd4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '114bcf4dd21e4f64980c9e41cfb4dab5'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:19.451183 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:19.457620 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-17db52b4-d67b-4e7c-971a-e56f3b33fbd4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '114bcf4dd21e4f64980c9e41cfb4dab5'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:19.476934 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1072/2120] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 00:57:19 2022] DELETE /identity/v3/users/114bcf4dd21e4f64980c9e41cfb4dab5 => generated 0 bytes in 254 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:19.480015 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17db52b4-d67b-4e7c-971a-e56f3b33fbd4 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:19.480015 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-17db52b4-d67b-4e7c-971a-e56f3b33fbd4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:19.508165 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb89fb15-024c-404e-93ef-9c7ae7dccee4 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:19.555453 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17db52b4-d67b-4e7c-971a-e56f3b33fbd4 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:19.561186 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-546cbc9a-2a6e-4c8e-a9f2-4290b02eb604 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:19.561852 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-546cbc9a-2a6e-4c8e-a9f2-4290b02eb604 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:19.562080 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-546cbc9a-2a6e-4c8e-a9f2-4290b02eb604 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:19.562356 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-546cbc9a-2a6e-4c8e-a9f2-4290b02eb604 admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:19.562995 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-546cbc9a-2a6e-4c8e-a9f2-4290b02eb604 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:19.564960 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-546cbc9a-2a6e-4c8e-a9f2-4290b02eb604 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:19.566315 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-578f656a-bd83-4bbc-ab6d-36eb1d050a03 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3f48335668a4e8f934ac11ff66cd308', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:19.567137 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-578f656a-bd83-4bbc-ab6d-36eb1d050a03 service cinder] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:19.567426 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-578f656a-bd83-4bbc-ab6d-36eb1d050a03 service cinder] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:19.567920 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-578f656a-bd83-4bbc-ab6d-36eb1d050a03 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:19.570082 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-578f656a-bd83-4bbc-ab6d-36eb1d050a03 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:19.589345 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1073/2121] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:57:19 2022] POST /identity/v3/users => generated 293 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:57:19.593409 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-546cbc9a-2a6e-4c8e-a9f2-4290b02eb604 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:19.605681 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-546cbc9a-2a6e-4c8e-a9f2-4290b02eb604 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:19.611485 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5723309-bc43-49b9-ac63-e6088345d4cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:19.612164 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5723309-bc43-49b9-ac63-e6088345d4cc admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:19.612513 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5723309-bc43-49b9-ac63-e6088345d4cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:19.612795 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5723309-bc43-49b9-ac63-e6088345d4cc admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:19.614484 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5723309-bc43-49b9-ac63-e6088345d4cc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:19.615154 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5723309-bc43-49b9-ac63-e6088345d4cc admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:19.629213 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1074/2122] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:57:19 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 09 00:57:19.629759 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:19.630067 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5723309-bc43-49b9-ac63-e6088345d4cc admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:19.655793 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-578f656a-bd83-4bbc-ab6d-36eb1d050a03 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:19.657004 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5723309-bc43-49b9-ac63-e6088345d4cc admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:19.658258 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-578f656a-bd83-4bbc-ab6d-36eb1d050a03 service cinder] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:19.663962 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da55697d-acc3-4bca-8bd0-8497ef356341 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:19.664577 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da55697d-acc3-4bca-8bd0-8497ef356341 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:19.664861 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da55697d-acc3-4bca-8bd0-8497ef356341 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:19.665150 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da55697d-acc3-4bca-8bd0-8497ef356341 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:19.665828 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da55697d-acc3-4bca-8bd0-8497ef356341 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:19.667440 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da55697d-acc3-4bca-8bd0-8497ef356341 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:19.673423 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1049/2123] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:57:19 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 234 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:57:19.683180 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1075/2124] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:57:19 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 09 00:57:19.684621 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-578f656a-bd83-4bbc-ab6d-36eb1d050a03 service cinder] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:19.698632 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-578f656a-bd83-4bbc-ab6d-36eb1d050a03 service cinder] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:19.704191 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78f35cdc-7499-4b9c-af03-0245de7812d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:19.704860 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f35cdc-7499-4b9c-af03-0245de7812d7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:19.705137 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f35cdc-7499-4b9c-af03-0245de7812d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:19.705571 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f35cdc-7499-4b9c-af03-0245de7812d7 admin admin] PATH_INFO: `/v3/projects/0571ef472ec449f5bf821d4c27e43c90/users/8cac123f51ca4bd1b7ea70bd0a84396d/roles/588016933bab436d8b7c3ac76827356d` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:19.722927 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78f35cdc-7499-4b9c-af03-0245de7812d7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0571ef472ec449f5bf821d4c27e43c90, user_id=8cac123f51ca4bd1b7ea70bd0a84396d, role_id=588016933bab436d8b7c3ac76827356d)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:19.732386 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78f35cdc-7499-4b9c-af03-0245de7812d7 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:19.761180 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1050/2125] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:57:19 2022] PUT /identity/v3/projects/0571ef472ec449f5bf821d4c27e43c90/users/8cac123f51ca4bd1b7ea70bd0a84396d/roles/588016933bab436d8b7c3ac76827356d => generated 0 bytes in 77 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:19.772775 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f109ba3b-f1a4-4b23-b3d2-d224c59427a6 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:19.773320 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f109ba3b-f1a4-4b23-b3d2-d224c59427a6 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:19.773592 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f109ba3b-f1a4-4b23-b3d2-d224c59427a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:19.808380 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78f35cdc-7499-4b9c-af03-0245de7812d7 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:19.810888 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-78f35cdc-7499-4b9c-af03-0245de7812d7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:19.814785 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-f109ba3b-f1a4-4b23-b3d2-d224c59427a6 None None] MFA Rules not processed for user `8cac123f51ca4bd1b7ea70bd0a84396d`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:19.816674 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-f109ba3b-f1a4-4b23-b3d2-d224c59427a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:19.894444 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78f35cdc-7499-4b9c-af03-0245de7812d7 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:19.901066 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61a35e3e-e9c9-428f-a79c-05f7101a40d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:19.902112 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a35e3e-e9c9-428f-a79c-05f7101a40d5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:19.902559 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a35e3e-e9c9-428f-a79c-05f7101a40d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:19.903569 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a35e3e-e9c9-428f-a79c-05f7101a40d5 admin admin] PATH_INFO: `/v3/projects/93fd69ab23c54fb08cff7e8d89e7adbf` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:19.904794 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61a35e3e-e9c9-428f-a79c-05f7101a40d5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=93fd69ab23c54fb08cff7e8d89e7adbf)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:19.906981 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61a35e3e-e9c9-428f-a79c-05f7101a40d5 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:19.907786 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1076/2126] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:19 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:57:19.912972 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f109ba3b-f1a4-4b23-b3d2-d224c59427a6 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:19.927214 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f109ba3b-f1a4-4b23-b3d2-d224c59427a6 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:19.933138 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9767e968-e1fa-4944-89d9-086c47a7730d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:19.934320 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9767e968-e1fa-4944-89d9-086c47a7730d admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:19.934320 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9767e968-e1fa-4944-89d9-086c47a7730d admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:19.934531 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9767e968-e1fa-4944-89d9-086c47a7730d admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:19.935107 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9767e968-e1fa-4944-89d9-086c47a7730d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:19.937190 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9767e968-e1fa-4944-89d9-086c47a7730d admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:19.966257 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1077/2127] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 00:57:19 2022] POST /identity/v3/projects => generated 383 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:57:19.969536 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-61a35e3e-e9c9-428f-a79c-05f7101a40d5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '93fd69ab23c54fb08cff7e8d89e7adbf', 'request_id': 'req-61a35e3e-e9c9-428f-a79c-05f7101a40d5'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:19.970782 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9767e968-e1fa-4944-89d9-086c47a7730d admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:19.978390 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-61a35e3e-e9c9-428f-a79c-05f7101a40d5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '93fd69ab23c54fb08cff7e8d89e7adbf', 'request_id': 'req-61a35e3e-e9c9-428f-a79c-05f7101a40d5'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:19.986263 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-61a35e3e-e9c9-428f-a79c-05f7101a40d5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '93fd69ab23c54fb08cff7e8d89e7adbf', 'request_id': 'req-61a35e3e-e9c9-428f-a79c-05f7101a40d5'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:19.986438 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:DELETE=3,keystone:SELECT=93,keystone:INSERT=4 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:20.000762 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-61a35e3e-e9c9-428f-a79c-05f7101a40d5 admin admin] The token cache is being invalidate because project 93fd69ab23c54fb08cff7e8d89e7adbf was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81453) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 00:57:20.001052 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-61a35e3e-e9c9-428f-a79c-05f7101a40d5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:20.001704 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-61a35e3e-e9c9-428f-a79c-05f7101a40d5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:20.003667 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1051/2128] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 00:57:19 2022] DELETE /identity/v3/projects/93fd69ab23c54fb08cff7e8d89e7adbf => generated 0 bytes in 196 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 09 00:57:20.007916 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61a35e3e-e9c9-428f-a79c-05f7101a40d5 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:20.009643 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-61a35e3e-e9c9-428f-a79c-05f7101a40d5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:20.048473 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9767e968-e1fa-4944-89d9-086c47a7730d admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:20.053411 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58f4edd4-6b8b-4562-9d20-dae50b23e2f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:20.054022 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58f4edd4-6b8b-4562-9d20-dae50b23e2f0 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:20.054303 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58f4edd4-6b8b-4562-9d20-dae50b23e2f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:20.054581 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58f4edd4-6b8b-4562-9d20-dae50b23e2f0 admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:20.055245 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58f4edd4-6b8b-4562-9d20-dae50b23e2f0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:20.057333 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58f4edd4-6b8b-4562-9d20-dae50b23e2f0 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:20.069544 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:20.083629 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1078/2129] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:57:19 2022] POST /identity/v3/users => generated 292 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 09 00:57:20.084057 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58f4edd4-6b8b-4562-9d20-dae50b23e2f0 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:20.086127 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-58f4edd4-6b8b-4562-9d20-dae50b23e2f0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:20.093914 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61a35e3e-e9c9-428f-a79c-05f7101a40d5 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:20.098803 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58f4edd4-6b8b-4562-9d20-dae50b23e2f0 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:20.099970 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d7194fa-dfe8-4f22-8047-58ecbaa8d796 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:20.100614 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d7194fa-dfe8-4f22-8047-58ecbaa8d796 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:20.100888 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d7194fa-dfe8-4f22-8047-58ecbaa8d796 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:20.101158 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d7194fa-dfe8-4f22-8047-58ecbaa8d796 admin admin] PATH_INFO: `/v3/users/da1827fbf6544a9e877dcf224c873ed1` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:20.102280 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d7194fa-dfe8-4f22-8047-58ecbaa8d796 admin admin] RBAC: Authorizing `identity:delete_user(user_id=da1827fbf6544a9e877dcf224c873ed1)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:20.104394 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d7194fa-dfe8-4f22-8047-58ecbaa8d796 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:20.104557 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-693c66bf-0cfb-486b-80a6-06ce56a7d2e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:20.105152 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-693c66bf-0cfb-486b-80a6-06ce56a7d2e5 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:20.105488 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-693c66bf-0cfb-486b-80a6-06ce56a7d2e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:20.105743 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-693c66bf-0cfb-486b-80a6-06ce56a7d2e5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:20.106386 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-693c66bf-0cfb-486b-80a6-06ce56a7d2e5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:20.108053 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-693c66bf-0cfb-486b-80a6-06ce56a7d2e5 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:20.118223 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1079/2130] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:57:20 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:57:20.121860 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-693c66bf-0cfb-486b-80a6-06ce56a7d2e5 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:20.132470 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-693c66bf-0cfb-486b-80a6-06ce56a7d2e5 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:20.137731 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a8cd1d6-387e-49ed-bd25-06e456240314 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:20.138340 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a8cd1d6-387e-49ed-bd25-06e456240314 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:20.138740 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a8cd1d6-387e-49ed-bd25-06e456240314 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:20.139056 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a8cd1d6-387e-49ed-bd25-06e456240314 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:20.139836 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a8cd1d6-387e-49ed-bd25-06e456240314 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:20.141435 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a8cd1d6-387e-49ed-bd25-06e456240314 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:20.152697 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1080/2131] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:57:20 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 09 00:57:20.154293 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-4d7194fa-dfe8-4f22-8047-58ecbaa8d796 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'da1827fbf6544a9e877dcf224c873ed1'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:20.156921 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a8cd1d6-387e-49ed-bd25-06e456240314 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:20.162815 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-4d7194fa-dfe8-4f22-8047-58ecbaa8d796 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'da1827fbf6544a9e877dcf224c873ed1'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:20.163504 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-4d7194fa-dfe8-4f22-8047-58ecbaa8d796 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'da1827fbf6544a9e877dcf224c873ed1'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:20.167401 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a8cd1d6-387e-49ed-bd25-06e456240314 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:20.171523 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-4d7194fa-dfe8-4f22-8047-58ecbaa8d796 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'da1827fbf6544a9e877dcf224c873ed1'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:20.172893 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-158f8072-0bf9-48b3-a2eb-3c669d3c7dcd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:20.173550 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-158f8072-0bf9-48b3-a2eb-3c669d3c7dcd admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:20.173827 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-158f8072-0bf9-48b3-a2eb-3c669d3c7dcd admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:20.174107 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-158f8072-0bf9-48b3-a2eb-3c669d3c7dcd admin admin] PATH_INFO: `/v3/projects/8ddaf92644c048f39d70583218cec933/users/3430bd39961e487a89a9ba08aea8a43d/roles/588016933bab436d8b7c3ac76827356d` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:20.185027 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-4d7194fa-dfe8-4f22-8047-58ecbaa8d796 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'da1827fbf6544a9e877dcf224c873ed1'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:20.191544 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-158f8072-0bf9-48b3-a2eb-3c669d3c7dcd admin admin] RBAC: Authorizing `identity:create_grant(project_id=8ddaf92644c048f39d70583218cec933, user_id=3430bd39961e487a89a9ba08aea8a43d, role_id=588016933bab436d8b7c3ac76827356d)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:20.192028 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1052/2132] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 00:57:20 2022] DELETE /identity/v3/users/da1827fbf6544a9e877dcf224c873ed1 => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 09 00:57:20.194320 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-158f8072-0bf9-48b3-a2eb-3c669d3c7dcd admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:20.208224 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1081/2133] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:57:20 2022] PUT /identity/v3/projects/8ddaf92644c048f39d70583218cec933/users/3430bd39961e487a89a9ba08aea8a43d/roles/588016933bab436d8b7c3ac76827356d => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 09 00:57:20.218514 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6b89af4-3bfb-4ec5-9d01-6aa6d522cb07 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:20.218804 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6b89af4-3bfb-4ec5-9d01-6aa6d522cb07 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:20.219091 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6b89af4-3bfb-4ec5-9d01-6aa6d522cb07 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:20.248184 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-f6b89af4-3bfb-4ec5-9d01-6aa6d522cb07 None None] MFA Rules not processed for user `3430bd39961e487a89a9ba08aea8a43d`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:20.249569 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-f6b89af4-3bfb-4ec5-9d01-6aa6d522cb07 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:20.317468 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1053/2134] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:20 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:57:20.321914 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-158f8072-0bf9-48b3-a2eb-3c669d3c7dcd admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:20.323708 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-158f8072-0bf9-48b3-a2eb-3c669d3c7dcd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:20.430364 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-158f8072-0bf9-48b3-a2eb-3c669d3c7dcd admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:20.435538 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdf7817b-f0e3-4e42-aee8-6f985385b789 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:20.436116 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdf7817b-f0e3-4e42-aee8-6f985385b789 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:20.436400 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdf7817b-f0e3-4e42-aee8-6f985385b789 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:20.436968 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdf7817b-f0e3-4e42-aee8-6f985385b789 admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:20.437637 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdf7817b-f0e3-4e42-aee8-6f985385b789 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:20.439761 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdf7817b-f0e3-4e42-aee8-6f985385b789 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:20.474792 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1082/2135] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 00:57:20 2022] POST /identity/v3/projects => generated 385 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:57:20.478900 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6b89af4-3bfb-4ec5-9d01-6aa6d522cb07 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:20.493447 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6b89af4-3bfb-4ec5-9d01-6aa6d522cb07 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:20.502569 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdf7817b-f0e3-4e42-aee8-6f985385b789 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:20.504974 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-fdf7817b-f0e3-4e42-aee8-6f985385b789 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:20.579365 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c9f54bb-39e7-489c-a580-5210f3599c46 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:20.580079 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c9f54bb-39e7-489c-a580-5210f3599c46 admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:20.580361 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c9f54bb-39e7-489c-a580-5210f3599c46 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:20.580637 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c9f54bb-39e7-489c-a580-5210f3599c46 admin admin] PATH_INFO: `/v3/users` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:20.582156 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c9f54bb-39e7-489c-a580-5210f3599c46 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:20.584102 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c9f54bb-39e7-489c-a580-5210f3599c46 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:20.586066 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:20.593653 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:20.608311 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1054/2136] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:57:20 2022] POST /identity/v3/users => generated 299 bytes in 130 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:57:20.610904 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdf7817b-f0e3-4e42-aee8-6f985385b789 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:20.612423 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c9f54bb-39e7-489c-a580-5210f3599c46 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:20.615820 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62428b67-a64f-4d3a-b564-07e9d9b5cba1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:20.616449 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62428b67-a64f-4d3a-b564-07e9d9b5cba1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:20.616733 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62428b67-a64f-4d3a-b564-07e9d9b5cba1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:20.617011 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62428b67-a64f-4d3a-b564-07e9d9b5cba1 admin admin] PATH_INFO: `/v3/projects/ea6cd31ea35842bd9e36c7a3b66a1fb3` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:20.618132 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62428b67-a64f-4d3a-b564-07e9d9b5cba1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ea6cd31ea35842bd9e36c7a3b66a1fb3)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:20.620089 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62428b67-a64f-4d3a-b564-07e9d9b5cba1 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:20.622372 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c9f54bb-39e7-489c-a580-5210f3599c46 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:20.626971 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-253889e1-b60d-43a6-b2d6-c7b6f0070260 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:20.627553 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-253889e1-b60d-43a6-b2d6-c7b6f0070260 admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:20.628449 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-253889e1-b60d-43a6-b2d6-c7b6f0070260 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:20.628728 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-253889e1-b60d-43a6-b2d6-c7b6f0070260 admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:20.629355 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-253889e1-b60d-43a6-b2d6-c7b6f0070260 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:20.630921 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-253889e1-b60d-43a6-b2d6-c7b6f0070260 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:20.641211 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1055/2137] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:57:20 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 09 00:57:20.645504 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-253889e1-b60d-43a6-b2d6-c7b6f0070260 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:20.656861 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-253889e1-b60d-43a6-b2d6-c7b6f0070260 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:20.674175 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-62428b67-a64f-4d3a-b564-07e9d9b5cba1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ea6cd31ea35842bd9e36c7a3b66a1fb3', 'request_id': 'req-62428b67-a64f-4d3a-b564-07e9d9b5cba1'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:20.682142 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-62428b67-a64f-4d3a-b564-07e9d9b5cba1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ea6cd31ea35842bd9e36c7a3b66a1fb3', 'request_id': 'req-62428b67-a64f-4d3a-b564-07e9d9b5cba1'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:20.689019 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-62428b67-a64f-4d3a-b564-07e9d9b5cba1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ea6cd31ea35842bd9e36c7a3b66a1fb3', 'request_id': 'req-62428b67-a64f-4d3a-b564-07e9d9b5cba1'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:20.701518 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-62428b67-a64f-4d3a-b564-07e9d9b5cba1 admin admin] The token cache is being invalidate because project ea6cd31ea35842bd9e36c7a3b66a1fb3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 00:57:20.701973 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-62428b67-a64f-4d3a-b564-07e9d9b5cba1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:20.702710 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-62428b67-a64f-4d3a-b564-07e9d9b5cba1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:20.704446 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1083/2138] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 00:57:20 2022] DELETE /identity/v3/projects/ea6cd31ea35842bd9e36c7a3b66a1fb3 => generated 0 bytes in 202 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:20.710474 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b22046-617b-40c8-add8-e196838bbe78 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:20.710763 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b22046-617b-40c8-add8-e196838bbe78 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:20.711041 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b22046-617b-40c8-add8-e196838bbe78 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:20.718670 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3df726b5-eeb7-4d24-a5eb-9a96a69dc7d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:20.719332 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df726b5-eeb7-4d24-a5eb-9a96a69dc7d2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:20.719648 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df726b5-eeb7-4d24-a5eb-9a96a69dc7d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:20.719931 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df726b5-eeb7-4d24-a5eb-9a96a69dc7d2 admin admin] PATH_INFO: `/v3/projects/b03af7cb152a4c6ab87b38c24e1fe7af/users/3e6e6d692fe943c188f077ae75ce3062/roles/092036ee46ac427baf4d56468575f697` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:20.730882 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-c8b22046-617b-40c8-add8-e196838bbe78 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:20.732104 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-c8b22046-617b-40c8-add8-e196838bbe78 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:20.735062 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3df726b5-eeb7-4d24-a5eb-9a96a69dc7d2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b03af7cb152a4c6ab87b38c24e1fe7af, user_id=3e6e6d692fe943c188f077ae75ce3062, role_id=092036ee46ac427baf4d56468575f697)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:20.737794 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3df726b5-eeb7-4d24-a5eb-9a96a69dc7d2 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:20.740193 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1084/2139] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:57:20.749302 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f21f2b6b-1200-41da-bab8-c7bf9e217ea5 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:20.749594 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f21f2b6b-1200-41da-bab8-c7bf9e217ea5 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:20.749867 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f21f2b6b-1200-41da-bab8-c7bf9e217ea5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:20.752537 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1056/2140] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:57:20 2022] PUT /identity/v3/projects/b03af7cb152a4c6ab87b38c24e1fe7af/users/3e6e6d692fe943c188f077ae75ce3062/roles/092036ee46ac427baf4d56468575f697 => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:20.756882 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3df726b5-eeb7-4d24-a5eb-9a96a69dc7d2 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:20.757996 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-3df726b5-eeb7-4d24-a5eb-9a96a69dc7d2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:20.771953 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-f21f2b6b-1200-41da-bab8-c7bf9e217ea5 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:20.773224 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-f21f2b6b-1200-41da-bab8-c7bf9e217ea5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:20.832445 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3df726b5-eeb7-4d24-a5eb-9a96a69dc7d2 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:20.837097 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76668931-bd47-4456-8a18-f28913b4510a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:20.837706 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76668931-bd47-4456-8a18-f28913b4510a admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:20.838108 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76668931-bd47-4456-8a18-f28913b4510a admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:20.838402 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76668931-bd47-4456-8a18-f28913b4510a admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:20.839053 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76668931-bd47-4456-8a18-f28913b4510a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:20.840810 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76668931-bd47-4456-8a18-f28913b4510a admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:20.841616 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1085/2141] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 09 00:57:20.847084 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f21f2b6b-1200-41da-bab8-c7bf9e217ea5 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:20.852334 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1057/2142] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:57:20 2022] GET /identity/v3/roles => generated 1420 bytes in 96 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:57:20.856315 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76668931-bd47-4456-8a18-f28913b4510a admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:20.859694 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f21f2b6b-1200-41da-bab8-c7bf9e217ea5 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:20.864964 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db41ccb5-982a-44a8-b8fa-0443a0b39582 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:20.865919 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db41ccb5-982a-44a8-b8fa-0443a0b39582 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:20.866116 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76668931-bd47-4456-8a18-f28913b4510a admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:20.866604 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db41ccb5-982a-44a8-b8fa-0443a0b39582 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:20.867071 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db41ccb5-982a-44a8-b8fa-0443a0b39582 admin admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:20.867955 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db41ccb5-982a-44a8-b8fa-0443a0b39582 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:57:20.868350 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db41ccb5-982a-44a8-b8fa-0443a0b39582 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:20.870059 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db41ccb5-982a-44a8-b8fa-0443a0b39582 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:20.870704 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2449e308-fdcf-404e-905e-450dc416e355 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:20.871346 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2449e308-fdcf-404e-905e-450dc416e355 admin admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:20.871650 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2449e308-fdcf-404e-905e-450dc416e355 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:20.871931 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2449e308-fdcf-404e-905e-450dc416e355 admin admin] PATH_INFO: `/v3/projects/b03af7cb152a4c6ab87b38c24e1fe7af/users/3e6e6d692fe943c188f077ae75ce3062/roles/092036ee46ac427baf4d56468575f697` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:20.873877 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2449e308-fdcf-404e-905e-450dc416e355 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b03af7cb152a4c6ab87b38c24e1fe7af, user_id=3e6e6d692fe943c188f077ae75ce3062, role_id=092036ee46ac427baf4d56468575f697)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:20.876550 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2449e308-fdcf-404e-905e-450dc416e355 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:20.886745 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1086/2143] 10.222.0.33 () {58 vars in 1153 bytes} [Sun Oct 9 00:57:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:57:20.887111 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1058/2144] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:57:20 2022] PUT /identity/v3/projects/b03af7cb152a4c6ab87b38c24e1fe7af/users/3e6e6d692fe943c188f077ae75ce3062/roles/092036ee46ac427baf4d56468575f697 => generated 0 bytes in 31 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 09 00:57:20.887843 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db41ccb5-982a-44a8-b8fa-0443a0b39582 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:20.897580 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ac866c0-5ca0-4234-9203-31acc273d4cf None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:20.897983 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ac866c0-5ca0-4234-9203-31acc273d4cf None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:20.898161 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db41ccb5-982a-44a8-b8fa-0443a0b39582 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:20.898264 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ac866c0-5ca0-4234-9203-31acc273d4cf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:20.928803 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-3ac866c0-5ca0-4234-9203-31acc273d4cf None None] MFA Rules not processed for user `3e6e6d692fe943c188f077ae75ce3062`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:20.930161 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-3ac866c0-5ca0-4234-9203-31acc273d4cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:20.945718 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:20.958046 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d90888b5-cd37-4edf-9280-773bdd518fa6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:20.958647 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d90888b5-cd37-4edf-9280-773bdd518fa6 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:20.958929 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d90888b5-cd37-4edf-9280-773bdd518fa6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:20.959206 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d90888b5-cd37-4edf-9280-773bdd518fa6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:20.959880 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d90888b5-cd37-4edf-9280-773bdd518fa6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:20.961775 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d90888b5-cd37-4edf-9280-773bdd518fa6 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:20.964528 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:INSERT=4,keystone:SELECT=96 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:20.989969 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1087/2145] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 00:57:20 2022] POST /identity/v3/projects => generated 377 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 09 00:57:20.995334 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d90888b5-cd37-4edf-9280-773bdd518fa6 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:21.004513 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1059/2146] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:20 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 09 00:57:21.015795 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ac866c0-5ca0-4234-9203-31acc273d4cf None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:21.017250 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-3ac866c0-5ca0-4234-9203-31acc273d4cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:21.061780 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d90888b5-cd37-4edf-9280-773bdd518fa6 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:21.067228 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb8940d-b685-436e-becd-dac177b42444 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:21.067889 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb8940d-b685-436e-becd-dac177b42444 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:21.068173 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb8940d-b685-436e-becd-dac177b42444 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:21.068448 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb8940d-b685-436e-becd-dac177b42444 admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:21.069080 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cb8940d-b685-436e-becd-dac177b42444 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:21.070999 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cb8940d-b685-436e-becd-dac177b42444 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:21.076082 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ac866c0-5ca0-4234-9203-31acc273d4cf None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:21.097685 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1088/2147] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:57:20 2022] POST /identity/v3/users => generated 289 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 09 00:57:21.098732 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb8940d-b685-436e-becd-dac177b42444 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:21.108259 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb8940d-b685-436e-becd-dac177b42444 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:21.112945 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acc58c2e-3605-471f-96ad-997d599f17c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:21.113526 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc58c2e-3605-471f-96ad-997d599f17c6 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:21.113806 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc58c2e-3605-471f-96ad-997d599f17c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:21.114081 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc58c2e-3605-471f-96ad-997d599f17c6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:21.114718 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acc58c2e-3605-471f-96ad-997d599f17c6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:21.116360 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acc58c2e-3605-471f-96ad-997d599f17c6 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:21.129720 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1089/2148] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:57:21 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:57:21.129875 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f63719e-608a-4791-9809-5fbd9ec69c06 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3f48335668a4e8f934ac11ff66cd308', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:21.129875 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f63719e-608a-4791-9809-5fbd9ec69c06 service cinder] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:21.129875 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f63719e-608a-4791-9809-5fbd9ec69c06 service cinder] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:21.129875 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f63719e-608a-4791-9809-5fbd9ec69c06 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:21.130224 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-5f63719e-608a-4791-9809-5fbd9ec69c06 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:21.130683 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acc58c2e-3605-471f-96ad-997d599f17c6 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:21.140114 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acc58c2e-3605-471f-96ad-997d599f17c6 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:21.145343 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a56ebee-e1a0-4591-ae94-ed8cb88eed10 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:21.145913 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a56ebee-e1a0-4591-ae94-ed8cb88eed10 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:21.146197 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a56ebee-e1a0-4591-ae94-ed8cb88eed10 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:21.146475 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a56ebee-e1a0-4591-ae94-ed8cb88eed10 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:21.147098 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a56ebee-e1a0-4591-ae94-ed8cb88eed10 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:21.148715 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a56ebee-e1a0-4591-ae94-ed8cb88eed10 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:21.159673 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1090/2149] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:57:21 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:57:21.163573 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a56ebee-e1a0-4591-ae94-ed8cb88eed10 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:21.174008 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a56ebee-e1a0-4591-ae94-ed8cb88eed10 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:21.179383 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b00d2ecc-8ada-4497-b826-62567bd59c59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:21.180119 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b00d2ecc-8ada-4497-b826-62567bd59c59 admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:21.180402 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b00d2ecc-8ada-4497-b826-62567bd59c59 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:21.180686 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b00d2ecc-8ada-4497-b826-62567bd59c59 admin admin] PATH_INFO: `/v3/projects/3ffe1ca49c4140ea8b6eecc9d2667b3b/users/ae420a548f06474db6eb4ca083883a34/roles/588016933bab436d8b7c3ac76827356d` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:21.195304 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f63719e-608a-4791-9809-5fbd9ec69c06 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:21.196056 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b00d2ecc-8ada-4497-b826-62567bd59c59 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3ffe1ca49c4140ea8b6eecc9d2667b3b, user_id=ae420a548f06474db6eb4ca083883a34, role_id=588016933bab436d8b7c3ac76827356d)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:21.196872 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f63719e-608a-4791-9809-5fbd9ec69c06 service cinder] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:21.198746 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b00d2ecc-8ada-4497-b826-62567bd59c59 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:21.206336 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1060/2150] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:57:21 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 191 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 09 00:57:21.211768 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1091/2151] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:57:21 2022] PUT /identity/v3/projects/3ffe1ca49c4140ea8b6eecc9d2667b3b/users/ae420a548f06474db6eb4ca083883a34/roles/588016933bab436d8b7c3ac76827356d => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 09 00:57:21.224001 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0577ed72-16f0-4d41-8bc2-6945d6e919a0 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:21.224289 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0577ed72-16f0-4d41-8bc2-6945d6e919a0 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:21.224586 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0577ed72-16f0-4d41-8bc2-6945d6e919a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:21.252285 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-0577ed72-16f0-4d41-8bc2-6945d6e919a0 None None] MFA Rules not processed for user `ae420a548f06474db6eb4ca083883a34`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:21.253545 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-0577ed72-16f0-4d41-8bc2-6945d6e919a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:21.283827 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:21.318551 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1061/2152] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:21 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:57:21.445775 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b00d2ecc-8ada-4497-b826-62567bd59c59 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:21.511553 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b00d2ecc-8ada-4497-b826-62567bd59c59 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:21.516649 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb28a884-e40b-4666-bcf1-7a4356df145a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3f48335668a4e8f934ac11ff66cd308', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:21.517214 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb28a884-e40b-4666-bcf1-7a4356df145a service cinder] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:21.517486 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb28a884-e40b-4666-bcf1-7a4356df145a service cinder] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:21.517758 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb28a884-e40b-4666-bcf1-7a4356df145a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:21.527146 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb28a884-e40b-4666-bcf1-7a4356df145a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:21.528500 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb28a884-e40b-4666-bcf1-7a4356df145a service cinder] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:21.547979 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1092/2153] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:57:21 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:57:22.021547 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0577ed72-16f0-4d41-8bc2-6945d6e919a0 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:22.030899 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0577ed72-16f0-4d41-8bc2-6945d6e919a0 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:22.035223 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-065a835b-f81d-4350-847c-5e8f50bc55df service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3f48335668a4e8f934ac11ff66cd308', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:22.035869 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-065a835b-f81d-4350-847c-5e8f50bc55df service cinder] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:22.036146 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-065a835b-f81d-4350-847c-5e8f50bc55df service cinder] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:22.036417 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-065a835b-f81d-4350-847c-5e8f50bc55df service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:22.038113 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-065a835b-f81d-4350-847c-5e8f50bc55df service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:22.102742 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-065a835b-f81d-4350-847c-5e8f50bc55df service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:22.104215 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-065a835b-f81d-4350-847c-5e8f50bc55df service cinder] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:22.114225 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1062/2154] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:57:22 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:57:22.996143 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb28a884-e40b-4666-bcf1-7a4356df145a service cinder] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:22.997583 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-fb28a884-e40b-4666-bcf1-7a4356df145a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:23.011423 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb28a884-e40b-4666-bcf1-7a4356df145a service cinder] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:23.026793 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddf2cdc9-0c79-48d6-a00b-7686f288d526 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3f48335668a4e8f934ac11ff66cd308', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:23.027637 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf2cdc9-0c79-48d6-a00b-7686f288d526 service cinder] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:23.028010 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf2cdc9-0c79-48d6-a00b-7686f288d526 service cinder] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:23.028353 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf2cdc9-0c79-48d6-a00b-7686f288d526 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:23.030535 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-ddf2cdc9-0c79-48d6-a00b-7686f288d526 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:23.052913 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:23.102993 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddf2cdc9-0c79-48d6-a00b-7686f288d526 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:23.105050 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddf2cdc9-0c79-48d6-a00b-7686f288d526 service cinder] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:23.115422 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1093/2155] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:57:22 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 120 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:57:23.796322 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-065a835b-f81d-4350-847c-5e8f50bc55df service cinder] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:23.808928 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-065a835b-f81d-4350-847c-5e8f50bc55df service cinder] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:23.812703 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e01b5c99-9e04-476f-a5bb-ff4fd072fc66 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3f48335668a4e8f934ac11ff66cd308', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:23.813305 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e01b5c99-9e04-476f-a5bb-ff4fd072fc66 service cinder] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:23.813621 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e01b5c99-9e04-476f-a5bb-ff4fd072fc66 service cinder] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:23.813899 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e01b5c99-9e04-476f-a5bb-ff4fd072fc66 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:23.879277 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e01b5c99-9e04-476f-a5bb-ff4fd072fc66 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:23.880967 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e01b5c99-9e04-476f-a5bb-ff4fd072fc66 service cinder] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:23.891648 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1063/2156] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:57:23 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:57:28.134447 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddf2cdc9-0c79-48d6-a00b-7686f288d526 service cinder] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:28.136918 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-ddf2cdc9-0c79-48d6-a00b-7686f288d526 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:28.228221 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddf2cdc9-0c79-48d6-a00b-7686f288d526 service cinder] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:28.233106 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48fd13be-100d-4023-8fc8-4d8d48db08b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:28.233824 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48fd13be-100d-4023-8fc8-4d8d48db08b3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:28.234115 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48fd13be-100d-4023-8fc8-4d8d48db08b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:28.234539 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48fd13be-100d-4023-8fc8-4d8d48db08b3 admin admin] PATH_INFO: `/v3/users/73bdf78202504dbe8356b731c7c1c57b` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:28.235702 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48fd13be-100d-4023-8fc8-4d8d48db08b3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=73bdf78202504dbe8356b731c7c1c57b)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:28.239088 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48fd13be-100d-4023-8fc8-4d8d48db08b3 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:28.298119 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-48fd13be-100d-4023-8fc8-4d8d48db08b3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '73bdf78202504dbe8356b731c7c1c57b'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:28.305528 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-48fd13be-100d-4023-8fc8-4d8d48db08b3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '73bdf78202504dbe8356b731c7c1c57b'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:28.306387 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-48fd13be-100d-4023-8fc8-4d8d48db08b3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '73bdf78202504dbe8356b731c7c1c57b'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:28.307665 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-48fd13be-100d-4023-8fc8-4d8d48db08b3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '73bdf78202504dbe8356b731c7c1c57b'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:28.317925 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-48fd13be-100d-4023-8fc8-4d8d48db08b3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '73bdf78202504dbe8356b731c7c1c57b'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:28.345246 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1094/2157] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 00:57:28 2022] DELETE /identity/v3/users/73bdf78202504dbe8356b731c7c1c57b => generated 0 bytes in 211 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:28.352266 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e17d1d3b-b43c-479f-aa24-e59cd245499c None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:28.352779 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e17d1d3b-b43c-479f-aa24-e59cd245499c None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:28.353279 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e17d1d3b-b43c-479f-aa24-e59cd245499c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:28.375614 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-e17d1d3b-b43c-479f-aa24-e59cd245499c None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:28.376958 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-e17d1d3b-b43c-479f-aa24-e59cd245499c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:28.441268 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:28.458121 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1064/2158] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:57:28.468539 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48fd13be-100d-4023-8fc8-4d8d48db08b3 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:28.470762 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-48fd13be-100d-4023-8fc8-4d8d48db08b3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:28.535888 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:28.552233 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48fd13be-100d-4023-8fc8-4d8d48db08b3 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:28.556876 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f419b73c-98f6-4401-8c09-822109893ee2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:28.557501 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f419b73c-98f6-4401-8c09-822109893ee2 service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:28.557797 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f419b73c-98f6-4401-8c09-822109893ee2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:28.558049 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f419b73c-98f6-4401-8c09-822109893ee2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:28.568971 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f419b73c-98f6-4401-8c09-822109893ee2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:28.570426 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f419b73c-98f6-4401-8c09-822109893ee2 service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:28.581171 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1095/2159] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:57:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 09 00:57:29.010758 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e17d1d3b-b43c-479f-aa24-e59cd245499c None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:29.012201 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-e17d1d3b-b43c-479f-aa24-e59cd245499c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:29.023874 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e17d1d3b-b43c-479f-aa24-e59cd245499c None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:29.028534 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11248606-814d-4db8-8523-1eb3e045f37c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:29.029159 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11248606-814d-4db8-8523-1eb3e045f37c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:29.029436 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11248606-814d-4db8-8523-1eb3e045f37c admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:29.029707 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11248606-814d-4db8-8523-1eb3e045f37c admin admin] PATH_INFO: `/v3/projects/91d97c3bee2d48ecac465ead2ab9b24a` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:29.030809 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11248606-814d-4db8-8523-1eb3e045f37c admin admin] RBAC: Authorizing `identity:delete_project(project_id=91d97c3bee2d48ecac465ead2ab9b24a)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:29.033008 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11248606-814d-4db8-8523-1eb3e045f37c admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:29.094687 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-11248606-814d-4db8-8523-1eb3e045f37c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '91d97c3bee2d48ecac465ead2ab9b24a', 'request_id': 'req-11248606-814d-4db8-8523-1eb3e045f37c'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:29.101379 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-11248606-814d-4db8-8523-1eb3e045f37c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '91d97c3bee2d48ecac465ead2ab9b24a', 'request_id': 'req-11248606-814d-4db8-8523-1eb3e045f37c'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:29.108228 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-11248606-814d-4db8-8523-1eb3e045f37c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '91d97c3bee2d48ecac465ead2ab9b24a', 'request_id': 'req-11248606-814d-4db8-8523-1eb3e045f37c'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:29.128017 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-11248606-814d-4db8-8523-1eb3e045f37c admin admin] The token cache is being invalidate because project 91d97c3bee2d48ecac465ead2ab9b24a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81453) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 00:57:29.128327 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-11248606-814d-4db8-8523-1eb3e045f37c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:29.129627 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-11248606-814d-4db8-8523-1eb3e045f37c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:29.134550 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1065/2160] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 00:57:29 2022] DELETE /identity/v3/projects/91d97c3bee2d48ecac465ead2ab9b24a => generated 0 bytes in 121 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:29.136242 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7267db02-4731-4db7-bb45-727ca8d35612 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:29.136686 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7267db02-4731-4db7-bb45-727ca8d35612 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:29.136827 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7267db02-4731-4db7-bb45-727ca8d35612 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:29.158741 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-7267db02-4731-4db7-bb45-727ca8d35612 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:29.160056 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-7267db02-4731-4db7-bb45-727ca8d35612 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:29.231751 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1096/2161] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 09 00:57:29.237558 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7805176c-c93a-4577-b954-e0a37878fee8 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:29.238618 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7805176c-c93a-4577-b954-e0a37878fee8 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:29.239102 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7805176c-c93a-4577-b954-e0a37878fee8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:29.263853 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-7805176c-c93a-4577-b954-e0a37878fee8 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:29.265370 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-7805176c-c93a-4577-b954-e0a37878fee8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:29.275638 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1066/2162] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:57:29.277830 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7267db02-4731-4db7-bb45-727ca8d35612 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:29.290028 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7267db02-4731-4db7-bb45-727ca8d35612 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:29.297483 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7af82ce-223f-4b6e-b598-581c3f01a993 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:29.298227 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7af82ce-223f-4b6e-b598-581c3f01a993 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:29.298513 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7af82ce-223f-4b6e-b598-581c3f01a993 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:29.298785 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7af82ce-223f-4b6e-b598-581c3f01a993 admin admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:29.299439 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7af82ce-223f-4b6e-b598-581c3f01a993 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:57:29.299798 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7af82ce-223f-4b6e-b598-581c3f01a993 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:29.301966 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7af82ce-223f-4b6e-b598-581c3f01a993 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:29.314182 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1097/2163] 10.222.0.33 () {58 vars in 1153 bytes} [Sun Oct 9 00:57:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:57:29.318560 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7805176c-c93a-4577-b954-e0a37878fee8 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:29.327702 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7805176c-c93a-4577-b954-e0a37878fee8 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:29.332387 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6cfc5fc-268d-4982-8b6c-bdcc437a8c54 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:29.332987 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6cfc5fc-268d-4982-8b6c-bdcc437a8c54 admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:29.333267 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6cfc5fc-268d-4982-8b6c-bdcc437a8c54 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:29.333538 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6cfc5fc-268d-4982-8b6c-bdcc437a8c54 admin admin] PATH_INFO: `/v3/projects` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:29.334167 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6cfc5fc-268d-4982-8b6c-bdcc437a8c54 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:29.336100 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6cfc5fc-268d-4982-8b6c-bdcc437a8c54 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:29.363974 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1067/2164] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 00:57:29 2022] POST /identity/v3/projects => generated 361 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:57:29.368437 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7af82ce-223f-4b6e-b598-581c3f01a993 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:29.379017 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7af82ce-223f-4b6e-b598-581c3f01a993 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:29.438169 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96a4304a-2077-4dc7-89f5-0b7ca90e22ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:29.438793 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96a4304a-2077-4dc7-89f5-0b7ca90e22ec admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:29.439077 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96a4304a-2077-4dc7-89f5-0b7ca90e22ec admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:29.439353 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96a4304a-2077-4dc7-89f5-0b7ca90e22ec admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:29.440020 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96a4304a-2077-4dc7-89f5-0b7ca90e22ec admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:29.442042 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96a4304a-2077-4dc7-89f5-0b7ca90e22ec admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:29.463177 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1098/2165] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:57:29 2022] POST /identity/v3/users => generated 281 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:57:29.467327 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6cfc5fc-268d-4982-8b6c-bdcc437a8c54 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:29.478169 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6cfc5fc-268d-4982-8b6c-bdcc437a8c54 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:29.482761 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16691471-8af6-4ecb-98d5-9ba09094ec66 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:29.483423 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16691471-8af6-4ecb-98d5-9ba09094ec66 admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:29.483802 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16691471-8af6-4ecb-98d5-9ba09094ec66 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:29.484147 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16691471-8af6-4ecb-98d5-9ba09094ec66 admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:29.484863 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16691471-8af6-4ecb-98d5-9ba09094ec66 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:29.486447 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16691471-8af6-4ecb-98d5-9ba09094ec66 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:29.497755 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1068/2166] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:57:29 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:57:29.502338 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96a4304a-2077-4dc7-89f5-0b7ca90e22ec admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:29.512511 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96a4304a-2077-4dc7-89f5-0b7ca90e22ec admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:29.517176 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2175be73-0e90-47ae-be61-a8189e081bf4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:29.517707 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2175be73-0e90-47ae-be61-a8189e081bf4 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:29.517988 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2175be73-0e90-47ae-be61-a8189e081bf4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:29.518259 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2175be73-0e90-47ae-be61-a8189e081bf4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:29.518892 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2175be73-0e90-47ae-be61-a8189e081bf4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:29.520588 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2175be73-0e90-47ae-be61-a8189e081bf4 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:29.533834 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1099/2167] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:57:29 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:57:29.538410 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16691471-8af6-4ecb-98d5-9ba09094ec66 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:29.549243 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16691471-8af6-4ecb-98d5-9ba09094ec66 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:29.554940 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd1422ec-2e3b-4291-9bee-1b33674c255b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:29.555717 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd1422ec-2e3b-4291-9bee-1b33674c255b admin admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:29.556061 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd1422ec-2e3b-4291-9bee-1b33674c255b admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:29.556411 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd1422ec-2e3b-4291-9bee-1b33674c255b admin admin] PATH_INFO: `/v3/projects/ffd9810571304dcd9c69dbc1b8e89763/users/b205306e6f8445aabddacd5b8c30221b/roles/588016933bab436d8b7c3ac76827356d` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:29.574131 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd1422ec-2e3b-4291-9bee-1b33674c255b admin admin] RBAC: Authorizing `identity:create_grant(project_id=ffd9810571304dcd9c69dbc1b8e89763, user_id=b205306e6f8445aabddacd5b8c30221b, role_id=588016933bab436d8b7c3ac76827356d)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:29.576797 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd1422ec-2e3b-4291-9bee-1b33674c255b admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:29.593876 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1069/2168] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:57:29 2022] PUT /identity/v3/projects/ffd9810571304dcd9c69dbc1b8e89763/users/b205306e6f8445aabddacd5b8c30221b/roles/588016933bab436d8b7c3ac76827356d => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:29.615983 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0027007a-4119-457b-8203-56313afe05e4 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:29.615983 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0027007a-4119-457b-8203-56313afe05e4 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:29.615983 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0027007a-4119-457b-8203-56313afe05e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:29.643273 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-0027007a-4119-457b-8203-56313afe05e4 None None] MFA Rules not processed for user `b205306e6f8445aabddacd5b8c30221b`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:29.644551 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-0027007a-4119-457b-8203-56313afe05e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:29.690810 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:29.726138 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1100/2169] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:29 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:57:29.761612 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd1422ec-2e3b-4291-9bee-1b33674c255b admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:29.763446 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-bd1422ec-2e3b-4291-9bee-1b33674c255b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:29.830000 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd1422ec-2e3b-4291-9bee-1b33674c255b admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:29.833934 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20be5842-a4b1-4d76-ba9f-c5ec1a1cdc6b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3f48335668a4e8f934ac11ff66cd308', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:29.834549 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20be5842-a4b1-4d76-ba9f-c5ec1a1cdc6b service cinder] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:29.834830 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20be5842-a4b1-4d76-ba9f-c5ec1a1cdc6b service cinder] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:29.835103 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20be5842-a4b1-4d76-ba9f-c5ec1a1cdc6b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:29.845395 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20be5842-a4b1-4d76-ba9f-c5ec1a1cdc6b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:29.847052 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20be5842-a4b1-4d76-ba9f-c5ec1a1cdc6b service cinder] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:29.857095 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1070/2170] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:57:29 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:57:29.872680 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0027007a-4119-457b-8203-56313afe05e4 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:29.953351 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0027007a-4119-457b-8203-56313afe05e4 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:29.958037 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-524bc2fe-f028-4b11-92a5-e583b04eb6a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:29.958669 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-524bc2fe-f028-4b11-92a5-e583b04eb6a9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:29.958945 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-524bc2fe-f028-4b11-92a5-e583b04eb6a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:29.959214 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-524bc2fe-f028-4b11-92a5-e583b04eb6a9 admin admin] PATH_INFO: `/v3/users/b205306e6f8445aabddacd5b8c30221b` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:29.961483 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-524bc2fe-f028-4b11-92a5-e583b04eb6a9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b205306e6f8445aabddacd5b8c30221b)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:29.963429 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-524bc2fe-f028-4b11-92a5-e583b04eb6a9 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:30.028343 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-524bc2fe-f028-4b11-92a5-e583b04eb6a9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b205306e6f8445aabddacd5b8c30221b'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:30.038218 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-524bc2fe-f028-4b11-92a5-e583b04eb6a9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b205306e6f8445aabddacd5b8c30221b'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:30.038574 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-524bc2fe-f028-4b11-92a5-e583b04eb6a9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b205306e6f8445aabddacd5b8c30221b'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:30.039180 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-524bc2fe-f028-4b11-92a5-e583b04eb6a9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b205306e6f8445aabddacd5b8c30221b'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:30.049314 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-524bc2fe-f028-4b11-92a5-e583b04eb6a9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b205306e6f8445aabddacd5b8c30221b'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:30.071803 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1101/2171] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 00:57:29 2022] DELETE /identity/v3/users/b205306e6f8445aabddacd5b8c30221b => generated 0 bytes in 199 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:30.076400 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4d72036-5e66-4e8c-a657-57105f4c924b None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:30.076747 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4d72036-5e66-4e8c-a657-57105f4c924b None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:30.076914 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4d72036-5e66-4e8c-a657-57105f4c924b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:30.098628 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-c4d72036-5e66-4e8c-a657-57105f4c924b None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:30.099933 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-c4d72036-5e66-4e8c-a657-57105f4c924b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:30.122308 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:30.185482 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1071/2172] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:57:30.195462 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-524bc2fe-f028-4b11-92a5-e583b04eb6a9 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:30.197051 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-524bc2fe-f028-4b11-92a5-e583b04eb6a9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:30.256074 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-524bc2fe-f028-4b11-92a5-e583b04eb6a9 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:30.260622 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1742808-311c-44df-9eac-ee94ab1931f0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:30.262802 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1742808-311c-44df-9eac-ee94ab1931f0 service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:30.262802 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1742808-311c-44df-9eac-ee94ab1931f0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:30.262802 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1742808-311c-44df-9eac-ee94ab1931f0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:30.271622 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1742808-311c-44df-9eac-ee94ab1931f0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:30.273055 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1742808-311c-44df-9eac-ee94ab1931f0 service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:30.280021 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:30.285792 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1102/2173] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:57:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:57:30.735712 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4d72036-5e66-4e8c-a657-57105f4c924b None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:30.739068 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-c4d72036-5e66-4e8c-a657-57105f4c924b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:30.751300 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4d72036-5e66-4e8c-a657-57105f4c924b None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:30.780540 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24b883b0-5111-4207-a2f1-cd6669f6a7e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:30.781267 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b883b0-5111-4207-a2f1-cd6669f6a7e7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:30.781617 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b883b0-5111-4207-a2f1-cd6669f6a7e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:30.782233 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b883b0-5111-4207-a2f1-cd6669f6a7e7 admin admin] PATH_INFO: `/v3/projects/ffd9810571304dcd9c69dbc1b8e89763` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:30.783464 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24b883b0-5111-4207-a2f1-cd6669f6a7e7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ffd9810571304dcd9c69dbc1b8e89763)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:30.786085 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24b883b0-5111-4207-a2f1-cd6669f6a7e7 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:30.843524 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-24b883b0-5111-4207-a2f1-cd6669f6a7e7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ffd9810571304dcd9c69dbc1b8e89763', 'request_id': 'req-24b883b0-5111-4207-a2f1-cd6669f6a7e7'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:30.852493 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-24b883b0-5111-4207-a2f1-cd6669f6a7e7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ffd9810571304dcd9c69dbc1b8e89763', 'request_id': 'req-24b883b0-5111-4207-a2f1-cd6669f6a7e7'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:30.860456 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-24b883b0-5111-4207-a2f1-cd6669f6a7e7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ffd9810571304dcd9c69dbc1b8e89763', 'request_id': 'req-24b883b0-5111-4207-a2f1-cd6669f6a7e7'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:30.877014 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-24b883b0-5111-4207-a2f1-cd6669f6a7e7 admin admin] The token cache is being invalidate because project ffd9810571304dcd9c69dbc1b8e89763 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81453) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 00:57:30.877523 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-24b883b0-5111-4207-a2f1-cd6669f6a7e7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:30.877988 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-24b883b0-5111-4207-a2f1-cd6669f6a7e7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:30.880542 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1072/2174] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 00:57:30 2022] DELETE /identity/v3/projects/ffd9810571304dcd9c69dbc1b8e89763 => generated 0 bytes in 145 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:30.887208 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a710d4ff-7150-4eb6-9b41-8089b400ea48 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:30.887856 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a710d4ff-7150-4eb6-9b41-8089b400ea48 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:30.888473 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a710d4ff-7150-4eb6-9b41-8089b400ea48 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:30.916381 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-a710d4ff-7150-4eb6-9b41-8089b400ea48 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:30.920461 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-a710d4ff-7150-4eb6-9b41-8089b400ea48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:31.012949 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1103/2175] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:57:31.021319 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ef2052-ecd7-475e-a1c0-21c855f6422a None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:31.022476 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ef2052-ecd7-475e-a1c0-21c855f6422a None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:31.023044 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ef2052-ecd7-475e-a1c0-21c855f6422a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:31.057824 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-f9ef2052-ecd7-475e-a1c0-21c855f6422a None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:31.059437 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-f9ef2052-ecd7-475e-a1c0-21c855f6422a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:31.075153 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1073/2176] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 52 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:57:31.076627 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a710d4ff-7150-4eb6-9b41-8089b400ea48 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:31.095500 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a710d4ff-7150-4eb6-9b41-8089b400ea48 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:31.102055 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63ab8702-0a78-4831-a8cd-3b38dcf97294 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:31.103012 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ab8702-0a78-4831-a8cd-3b38dcf97294 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:31.103296 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ab8702-0a78-4831-a8cd-3b38dcf97294 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:31.103568 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ab8702-0a78-4831-a8cd-3b38dcf97294 admin admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:31.104263 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63ab8702-0a78-4831-a8cd-3b38dcf97294 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:57:31.104562 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63ab8702-0a78-4831-a8cd-3b38dcf97294 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:31.106178 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63ab8702-0a78-4831-a8cd-3b38dcf97294 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:31.119790 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1104/2177] 10.222.0.33 () {58 vars in 1153 bytes} [Sun Oct 9 00:57:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:57:31.123728 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9ef2052-ecd7-475e-a1c0-21c855f6422a None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:31.134600 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9ef2052-ecd7-475e-a1c0-21c855f6422a None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:31.144341 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cfa8373-9843-4ef6-9539-701e21e7e24d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:31.145063 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cfa8373-9843-4ef6-9539-701e21e7e24d admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:31.145349 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cfa8373-9843-4ef6-9539-701e21e7e24d admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:31.145628 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cfa8373-9843-4ef6-9539-701e21e7e24d admin admin] PATH_INFO: `/v3/projects` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:31.146298 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cfa8373-9843-4ef6-9539-701e21e7e24d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:31.148445 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cfa8373-9843-4ef6-9539-701e21e7e24d admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:31.210683 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1074/2178] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 00:57:31 2022] POST /identity/v3/projects => generated 377 bytes in 86 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:57:31.213554 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63ab8702-0a78-4831-a8cd-3b38dcf97294 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:31.229187 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63ab8702-0a78-4831-a8cd-3b38dcf97294 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:31.339546 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c5204fe-bbd7-4247-8c8b-95e3c0164854 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:31.340366 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c5204fe-bbd7-4247-8c8b-95e3c0164854 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:31.340479 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c5204fe-bbd7-4247-8c8b-95e3c0164854 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:31.340741 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c5204fe-bbd7-4247-8c8b-95e3c0164854 admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:31.341578 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c5204fe-bbd7-4247-8c8b-95e3c0164854 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:31.344182 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c5204fe-bbd7-4247-8c8b-95e3c0164854 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:31.389435 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1105/2179] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:57:31 2022] POST /identity/v3/users => generated 289 bytes in 176 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:57:31.390637 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cfa8373-9843-4ef6-9539-701e21e7e24d admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:31.412425 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cfa8373-9843-4ef6-9539-701e21e7e24d admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:31.417453 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f872deb-c106-4e80-b2f0-dda7b89d037b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:31.418087 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f872deb-c106-4e80-b2f0-dda7b89d037b admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:31.418340 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f872deb-c106-4e80-b2f0-dda7b89d037b admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:31.418639 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f872deb-c106-4e80-b2f0-dda7b89d037b admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:31.419269 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f872deb-c106-4e80-b2f0-dda7b89d037b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:31.420909 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f872deb-c106-4e80-b2f0-dda7b89d037b admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:31.432682 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1075/2180] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:57:31 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 09 00:57:31.436707 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c5204fe-bbd7-4247-8c8b-95e3c0164854 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:31.451193 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c5204fe-bbd7-4247-8c8b-95e3c0164854 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:31.456917 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84ae432a-a28d-4165-b56e-93bbc0dd4fbb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:31.458161 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84ae432a-a28d-4165-b56e-93bbc0dd4fbb admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:31.458431 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84ae432a-a28d-4165-b56e-93bbc0dd4fbb admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:31.458966 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84ae432a-a28d-4165-b56e-93bbc0dd4fbb admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:31.459634 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84ae432a-a28d-4165-b56e-93bbc0dd4fbb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:31.461390 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84ae432a-a28d-4165-b56e-93bbc0dd4fbb admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:31.474582 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1106/2181] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:57:31 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:57:31.478896 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f872deb-c106-4e80-b2f0-dda7b89d037b admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:31.488840 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f872deb-c106-4e80-b2f0-dda7b89d037b admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:31.493484 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f00a7935-84fb-46fd-93f5-c153c410bd81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:31.494138 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f00a7935-84fb-46fd-93f5-c153c410bd81 admin admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:31.494444 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f00a7935-84fb-46fd-93f5-c153c410bd81 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:31.494705 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f00a7935-84fb-46fd-93f5-c153c410bd81 admin admin] PATH_INFO: `/v3/projects/28bc82345ea24b929b66fbcded2b97b7/users/04bc22d558c2466ba43a456d08bdd6bd/roles/588016933bab436d8b7c3ac76827356d` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:31.511823 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f00a7935-84fb-46fd-93f5-c153c410bd81 admin admin] RBAC: Authorizing `identity:create_grant(project_id=28bc82345ea24b929b66fbcded2b97b7, user_id=04bc22d558c2466ba43a456d08bdd6bd, role_id=588016933bab436d8b7c3ac76827356d)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:31.514923 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f00a7935-84fb-46fd-93f5-c153c410bd81 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:31.537338 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1076/2182] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:57:31 2022] PUT /identity/v3/projects/28bc82345ea24b929b66fbcded2b97b7/users/04bc22d558c2466ba43a456d08bdd6bd/roles/588016933bab436d8b7c3ac76827356d => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:31.541588 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2962f17e-2d6f-4fa3-bc48-0b65b3137bad None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:31.541928 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2962f17e-2d6f-4fa3-bc48-0b65b3137bad None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:31.542235 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2962f17e-2d6f-4fa3-bc48-0b65b3137bad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:31.563743 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-2962f17e-2d6f-4fa3-bc48-0b65b3137bad None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:31.564726 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-2962f17e-2d6f-4fa3-bc48-0b65b3137bad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:31.577352 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f00a7935-84fb-46fd-93f5-c153c410bd81 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:31.580022 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-f00a7935-84fb-46fd-93f5-c153c410bd81 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:31.630790 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1107/2183] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 09 00:57:31.632352 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273b8a55-e33a-433a-8976-6210a657d375 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:31.632708 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273b8a55-e33a-433a-8976-6210a657d375 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:31.633057 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273b8a55-e33a-433a-8976-6210a657d375 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:31.634767 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1108/2184] 10.222.0.33 () {54 vars in 921 bytes} [Sun Oct 9 00:57:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:57:31.637341 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:31.642247 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-273b8a55-e33a-433a-8976-6210a657d375 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:31.644999 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-273b8a55-e33a-433a-8976-6210a657d375 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:31.657191 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:31.660516 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f00a7935-84fb-46fd-93f5-c153c410bd81 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:31.666412 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fddec3f-6fa4-4e2d-b35a-147ad6e0f99d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:31.667054 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fddec3f-6fa4-4e2d-b35a-147ad6e0f99d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:31.667384 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fddec3f-6fa4-4e2d-b35a-147ad6e0f99d admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:31.667693 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fddec3f-6fa4-4e2d-b35a-147ad6e0f99d admin admin] PATH_INFO: `/v3/users/ee8c3cfe357c4d3183d3cba87646d289` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:31.668938 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fddec3f-6fa4-4e2d-b35a-147ad6e0f99d admin admin] RBAC: Authorizing `identity:delete_user(user_id=ee8c3cfe357c4d3183d3cba87646d289)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:31.671579 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fddec3f-6fa4-4e2d-b35a-147ad6e0f99d admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:31.732895 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-2fddec3f-6fa4-4e2d-b35a-147ad6e0f99d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ee8c3cfe357c4d3183d3cba87646d289'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:31.737090 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-273b8a55-e33a-433a-8976-6210a657d375 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:31.741011 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3cd6205-77ca-4fdc-b409-d89e32db5935 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:31.741172 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-2fddec3f-6fa4-4e2d-b35a-147ad6e0f99d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ee8c3cfe357c4d3183d3cba87646d289'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:31.741614 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3cd6205-77ca-4fdc-b409-d89e32db5935 service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:31.741894 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3cd6205-77ca-4fdc-b409-d89e32db5935 service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:31.742136 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3cd6205-77ca-4fdc-b409-d89e32db5935 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:31.742259 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-2fddec3f-6fa4-4e2d-b35a-147ad6e0f99d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ee8c3cfe357c4d3183d3cba87646d289'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:31.743986 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-b3cd6205-77ca-4fdc-b409-d89e32db5935 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:31.751203 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-2fddec3f-6fa4-4e2d-b35a-147ad6e0f99d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ee8c3cfe357c4d3183d3cba87646d289'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:31.758315 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3cd6205-77ca-4fdc-b409-d89e32db5935 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:31.761601 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3cd6205-77ca-4fdc-b409-d89e32db5935 service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:31.767299 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-2fddec3f-6fa4-4e2d-b35a-147ad6e0f99d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ee8c3cfe357c4d3183d3cba87646d289'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:31.772240 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1109/2185] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:57:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 131 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:57:31.773602 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-860fe586-7ed1-4f63-a5d7-690a3c2d5e45 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:31.773940 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-860fe586-7ed1-4f63-a5d7-690a3c2d5e45 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:31.774215 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-860fe586-7ed1-4f63-a5d7-690a3c2d5e45 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:31.777378 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1077/2186] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 00:57:31 2022] DELETE /identity/v3/users/ee8c3cfe357c4d3183d3cba87646d289 => generated 0 bytes in 197 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 09 00:57:31.778620 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fddec3f-6fa4-4e2d-b35a-147ad6e0f99d admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:31.780365 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-2fddec3f-6fa4-4e2d-b35a-147ad6e0f99d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:31.802837 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-860fe586-7ed1-4f63-a5d7-690a3c2d5e45 None None] MFA Rules not processed for user `d25e240cb93643ed8122ccac9586936f`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:31.804361 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-860fe586-7ed1-4f63-a5d7-690a3c2d5e45 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:31.881361 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fddec3f-6fa4-4e2d-b35a-147ad6e0f99d admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:31.908440 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2bf2b51-3f96-4dcc-91d4-53ff054fe28b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:31.909771 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2bf2b51-3f96-4dcc-91d4-53ff054fe28b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:31.910325 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2bf2b51-3f96-4dcc-91d4-53ff054fe28b admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:31.910890 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2bf2b51-3f96-4dcc-91d4-53ff054fe28b admin admin] PATH_INFO: `/v3/users/15e90d2d2b184f339d1d2aacb77bc82b` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:31.912714 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2bf2b51-3f96-4dcc-91d4-53ff054fe28b admin admin] RBAC: Authorizing `identity:delete_user(user_id=15e90d2d2b184f339d1d2aacb77bc82b)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:31.914461 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1110/2187] 10.222.0.33 () {58 vars in 1018 bytes} [Sun Oct 9 00:57:31 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:57:31.916468 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2bf2b51-3f96-4dcc-91d4-53ff054fe28b admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:31.927640 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c730d03a-b13d-4683-b47f-93d34ef025fb None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:31.928148 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c730d03a-b13d-4683-b47f-93d34ef025fb None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:31.928581 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c730d03a-b13d-4683-b47f-93d34ef025fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:31.971934 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-c730d03a-b13d-4683-b47f-93d34ef025fb None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:31.973612 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-c730d03a-b13d-4683-b47f-93d34ef025fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:31.995230 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1111/2188] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:57:31.996504 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19e15cb-a5c0-442f-b3de-4e9e8671c8cb None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:31.996781 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19e15cb-a5c0-442f-b3de-4e9e8671c8cb None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:31.997040 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19e15cb-a5c0-442f-b3de-4e9e8671c8cb None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:31.999692 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1112/2189] 10.222.0.33 () {54 vars in 961 bytes} [Sun Oct 9 00:57:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:57:32.009910 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c19e15cb-a5c0-442f-b3de-4e9e8671c8cb None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:32.013372 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-c19e15cb-a5c0-442f-b3de-4e9e8671c8cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:32.017458 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-c2bf2b51-3f96-4dcc-91d4-53ff054fe28b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '15e90d2d2b184f339d1d2aacb77bc82b'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:32.026557 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-c2bf2b51-3f96-4dcc-91d4-53ff054fe28b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '15e90d2d2b184f339d1d2aacb77bc82b'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:32.027183 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-c2bf2b51-3f96-4dcc-91d4-53ff054fe28b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '15e90d2d2b184f339d1d2aacb77bc82b'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:32.039154 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-c2bf2b51-3f96-4dcc-91d4-53ff054fe28b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '15e90d2d2b184f339d1d2aacb77bc82b'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:32.041070 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=80,keystone:DELETE=18,keystone:INSERT=2 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:32.058136 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-c2bf2b51-3f96-4dcc-91d4-53ff054fe28b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '15e90d2d2b184f339d1d2aacb77bc82b'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:32.066795 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1078/2190] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 00:57:31 2022] DELETE /identity/v3/users/15e90d2d2b184f339d1d2aacb77bc82b => generated 0 bytes in 289 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:32.068292 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b42809f-6132-4a71-8a9a-1a0f0b7b981a None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:32.068638 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b42809f-6132-4a71-8a9a-1a0f0b7b981a None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:32.068975 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b42809f-6132-4a71-8a9a-1a0f0b7b981a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:32.121319 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-5b42809f-6132-4a71-8a9a-1a0f0b7b981a None None] MFA Rules not processed for user `d25e240cb93643ed8122ccac9586936f`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:32.123855 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-5b42809f-6132-4a71-8a9a-1a0f0b7b981a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:32.132584 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c19e15cb-a5c0-442f-b3de-4e9e8671c8cb None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:32.263815 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1079/2191] 10.222.0.33 () {58 vars in 1058 bytes} [Sun Oct 9 00:57:32 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:57:32.267787 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39286d64-8172-4ee9-abf4-1db11c128d25 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:32.268409 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39286d64-8172-4ee9-abf4-1db11c128d25 service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:32.268692 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39286d64-8172-4ee9-abf4-1db11c128d25 service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:32.269027 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39286d64-8172-4ee9-abf4-1db11c128d25 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:32.271809 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-39286d64-8172-4ee9-abf4-1db11c128d25 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:32.274432 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b42809f-6132-4a71-8a9a-1a0f0b7b981a None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:32.290113 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b42809f-6132-4a71-8a9a-1a0f0b7b981a None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:32.316848 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:32.316944 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48c14ba7-718c-404f-887a-e03c9ffd3973 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd25e240cb93643ed8122ccac9586936f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:32.316944 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c14ba7-718c-404f-887a-e03c9ffd3973 service placement] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:32.316944 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c14ba7-718c-404f-887a-e03c9ffd3973 service placement] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:32.316944 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c14ba7-718c-404f-887a-e03c9ffd3973 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:32.317260 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-48c14ba7-718c-404f-887a-e03c9ffd3973 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:32.336592 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48c14ba7-718c-404f-887a-e03c9ffd3973 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:32.339146 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48c14ba7-718c-404f-887a-e03c9ffd3973 service placement] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:32.354404 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1080/2192] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:57:32 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 80 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:57:32.410029 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39286d64-8172-4ee9-abf4-1db11c128d25 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:32.411910 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39286d64-8172-4ee9-abf4-1db11c128d25 service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:32.422270 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1113/2193] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:57:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 413 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 09 00:57:32.726054 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48c14ba7-718c-404f-887a-e03c9ffd3973 service placement] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:32.727704 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-48c14ba7-718c-404f-887a-e03c9ffd3973 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:32.772627 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48c14ba7-718c-404f-887a-e03c9ffd3973 service placement] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:32.786759 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-572ae548-9cd7-4eaa-9261-4513e8a8404a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:32.787695 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-572ae548-9cd7-4eaa-9261-4513e8a8404a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:32.788582 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-572ae548-9cd7-4eaa-9261-4513e8a8404a admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:32.789007 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-572ae548-9cd7-4eaa-9261-4513e8a8404a admin admin] PATH_INFO: `/v3/projects/88e7a07d04d74e739b7fa17f73018b64` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:32.792619 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-572ae548-9cd7-4eaa-9261-4513e8a8404a admin admin] RBAC: Authorizing `identity:delete_project(project_id=88e7a07d04d74e739b7fa17f73018b64)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:32.794925 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-572ae548-9cd7-4eaa-9261-4513e8a8404a admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:32.869054 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-572ae548-9cd7-4eaa-9261-4513e8a8404a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '88e7a07d04d74e739b7fa17f73018b64', 'request_id': 'req-572ae548-9cd7-4eaa-9261-4513e8a8404a'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:32.894751 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-572ae548-9cd7-4eaa-9261-4513e8a8404a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '88e7a07d04d74e739b7fa17f73018b64', 'request_id': 'req-572ae548-9cd7-4eaa-9261-4513e8a8404a'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:32.905832 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-572ae548-9cd7-4eaa-9261-4513e8a8404a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '88e7a07d04d74e739b7fa17f73018b64', 'request_id': 'req-572ae548-9cd7-4eaa-9261-4513e8a8404a'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:32.920803 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-572ae548-9cd7-4eaa-9261-4513e8a8404a admin admin] The token cache is being invalidate because project 88e7a07d04d74e739b7fa17f73018b64 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81453) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 00:57:32.921100 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-572ae548-9cd7-4eaa-9261-4513e8a8404a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:32.921804 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-572ae548-9cd7-4eaa-9261-4513e8a8404a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:32.924958 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1081/2194] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 00:57:32 2022] DELETE /identity/v3/projects/88e7a07d04d74e739b7fa17f73018b64 => generated 0 bytes in 199 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:32.930083 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc6f7d4-5650-49c5-a688-fd02caf6a342 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:32.930818 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc6f7d4-5650-49c5-a688-fd02caf6a342 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:32.931192 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc6f7d4-5650-49c5-a688-fd02caf6a342 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:32.945475 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-572ae548-9cd7-4eaa-9261-4513e8a8404a admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:32.948150 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-572ae548-9cd7-4eaa-9261-4513e8a8404a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:32.963052 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-cdc6f7d4-5650-49c5-a688-fd02caf6a342 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:32.964827 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-cdc6f7d4-5650-49c5-a688-fd02caf6a342 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:33.005914 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:33.037624 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-572ae548-9cd7-4eaa-9261-4513e8a8404a admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:33.050280 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3984b162-17b1-4e12-90e1-93c072c2a39a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:33.051821 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3984b162-17b1-4e12-90e1-93c072c2a39a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:33.052408 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3984b162-17b1-4e12-90e1-93c072c2a39a admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:33.053428 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3984b162-17b1-4e12-90e1-93c072c2a39a admin admin] PATH_INFO: `/v3/projects/c15c3df3639a48dc9d1276290b1a0173` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:33.054926 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3984b162-17b1-4e12-90e1-93c072c2a39a admin admin] RBAC: Authorizing `identity:delete_project(project_id=c15c3df3639a48dc9d1276290b1a0173)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:33.065087 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3984b162-17b1-4e12-90e1-93c072c2a39a admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:33.101289 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1114/2195] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 09 00:57:33.112644 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dcdee59-0b34-4469-bcc5-cabbf1e24b44 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:33.112980 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dcdee59-0b34-4469-bcc5-cabbf1e24b44 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:33.114250 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dcdee59-0b34-4469-bcc5-cabbf1e24b44 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:33.146496 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-6dcdee59-0b34-4469-bcc5-cabbf1e24b44 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:33.148135 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-6dcdee59-0b34-4469-bcc5-cabbf1e24b44 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:33.151569 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-3984b162-17b1-4e12-90e1-93c072c2a39a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c15c3df3639a48dc9d1276290b1a0173', 'request_id': 'req-3984b162-17b1-4e12-90e1-93c072c2a39a'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:33.159361 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-3984b162-17b1-4e12-90e1-93c072c2a39a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c15c3df3639a48dc9d1276290b1a0173', 'request_id': 'req-3984b162-17b1-4e12-90e1-93c072c2a39a'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:33.168426 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-3984b162-17b1-4e12-90e1-93c072c2a39a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c15c3df3639a48dc9d1276290b1a0173', 'request_id': 'req-3984b162-17b1-4e12-90e1-93c072c2a39a'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:33.189277 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-3984b162-17b1-4e12-90e1-93c072c2a39a admin admin] The token cache is being invalidate because project c15c3df3639a48dc9d1276290b1a0173 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81453) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 00:57:33.189948 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-3984b162-17b1-4e12-90e1-93c072c2a39a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:33.190848 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-3984b162-17b1-4e12-90e1-93c072c2a39a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:33.193328 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1082/2196] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 00:57:32 2022] DELETE /identity/v3/projects/c15c3df3639a48dc9d1276290b1a0173 => generated 0 bytes in 253 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:33.197674 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3984b162-17b1-4e12-90e1-93c072c2a39a admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:33.199585 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-3984b162-17b1-4e12-90e1-93c072c2a39a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:33.231641 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1115/2197] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 09 00:57:33.236061 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dcdee59-0b34-4469-bcc5-cabbf1e24b44 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:33.246899 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dcdee59-0b34-4469-bcc5-cabbf1e24b44 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:33.254221 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6dc96cb-147c-45b2-ba4c-323d003aca34 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:33.255500 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6dc96cb-147c-45b2-ba4c-323d003aca34 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:33.255868 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6dc96cb-147c-45b2-ba4c-323d003aca34 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:33.256209 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6dc96cb-147c-45b2-ba4c-323d003aca34 admin admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:33.257499 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6dc96cb-147c-45b2-ba4c-323d003aca34 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:57:33.257876 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6dc96cb-147c-45b2-ba4c-323d003aca34 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:33.261293 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6dc96cb-147c-45b2-ba4c-323d003aca34 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:33.280003 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1116/2198] 10.222.0.33 () {58 vars in 1153 bytes} [Sun Oct 9 00:57:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:57:33.282724 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3984b162-17b1-4e12-90e1-93c072c2a39a admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:33.283865 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6dc96cb-147c-45b2-ba4c-323d003aca34 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:33.289680 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b359cb2-7fd4-46e5-8fea-09d4ff9334c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:33.290826 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b359cb2-7fd4-46e5-8fea-09d4ff9334c1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:33.291213 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b359cb2-7fd4-46e5-8fea-09d4ff9334c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:33.291490 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b359cb2-7fd4-46e5-8fea-09d4ff9334c1 admin admin] PATH_INFO: `/v3/users/65375e71d0d940658ef71f9f3f613ec0` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:33.294135 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b359cb2-7fd4-46e5-8fea-09d4ff9334c1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=65375e71d0d940658ef71f9f3f613ec0)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:33.296267 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b359cb2-7fd4-46e5-8fea-09d4ff9334c1 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:33.298962 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6dc96cb-147c-45b2-ba4c-323d003aca34 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:33.307098 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46a3d26f-fdfa-40ad-93ef-987882bba1b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:33.307856 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a3d26f-fdfa-40ad-93ef-987882bba1b1 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:33.308207 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a3d26f-fdfa-40ad-93ef-987882bba1b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:33.308551 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a3d26f-fdfa-40ad-93ef-987882bba1b1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:33.309269 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46a3d26f-fdfa-40ad-93ef-987882bba1b1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:33.311664 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46a3d26f-fdfa-40ad-93ef-987882bba1b1 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:33.371292 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1117/2199] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 00:57:33 2022] POST /identity/v3/projects => generated 381 bytes in 88 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:57:33.376013 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46a3d26f-fdfa-40ad-93ef-987882bba1b1 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:33.397762 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:33.413187 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-8b359cb2-7fd4-46e5-8fea-09d4ff9334c1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '65375e71d0d940658ef71f9f3f613ec0'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:33.435247 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-8b359cb2-7fd4-46e5-8fea-09d4ff9334c1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '65375e71d0d940658ef71f9f3f613ec0'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:33.436274 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-8b359cb2-7fd4-46e5-8fea-09d4ff9334c1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '65375e71d0d940658ef71f9f3f613ec0'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:33.448065 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-8b359cb2-7fd4-46e5-8fea-09d4ff9334c1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '65375e71d0d940658ef71f9f3f613ec0'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:33.466515 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:33.468991 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-8b359cb2-7fd4-46e5-8fea-09d4ff9334c1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '65375e71d0d940658ef71f9f3f613ec0'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:33.471366 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46a3d26f-fdfa-40ad-93ef-987882bba1b1 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:33.482074 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1083/2200] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 00:57:33 2022] DELETE /identity/v3/users/65375e71d0d940658ef71f9f3f613ec0 => generated 0 bytes in 282 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:33.482264 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd995318-ab2c-49a1-a5e8-44527e3effb7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:33.482427 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b359cb2-7fd4-46e5-8fea-09d4ff9334c1 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:33.482550 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd995318-ab2c-49a1-a5e8-44527e3effb7 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:33.483516 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-8b359cb2-7fd4-46e5-8fea-09d4ff9334c1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:33.491922 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd995318-ab2c-49a1-a5e8-44527e3effb7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:33.493496 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd995318-ab2c-49a1-a5e8-44527e3effb7 admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:33.493496 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd995318-ab2c-49a1-a5e8-44527e3effb7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:33.494899 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd995318-ab2c-49a1-a5e8-44527e3effb7 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:33.552399 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1118/2201] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:57:33 2022] POST /identity/v3/users => generated 291 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:57:33.553657 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd995318-ab2c-49a1-a5e8-44527e3effb7 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:33.555354 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-bd995318-ab2c-49a1-a5e8-44527e3effb7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:33.586300 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd995318-ab2c-49a1-a5e8-44527e3effb7 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:33.606852 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b359cb2-7fd4-46e5-8fea-09d4ff9334c1 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:33.616482 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39b1e281-0337-4b7a-b2c3-cc494fa5c22d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:33.618213 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39b1e281-0337-4b7a-b2c3-cc494fa5c22d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:33.618689 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39b1e281-0337-4b7a-b2c3-cc494fa5c22d admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:33.619761 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39b1e281-0337-4b7a-b2c3-cc494fa5c22d admin admin] PATH_INFO: `/v3/users/8cac123f51ca4bd1b7ea70bd0a84396d` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:33.621334 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39b1e281-0337-4b7a-b2c3-cc494fa5c22d admin admin] RBAC: Authorizing `identity:delete_user(user_id=8cac123f51ca4bd1b7ea70bd0a84396d)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:33.623522 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39b1e281-0337-4b7a-b2c3-cc494fa5c22d admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:33.717284 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63ae1746-a68a-4f33-812c-b68315e0409a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:33.720822 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ae1746-a68a-4f33-812c-b68315e0409a admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:33.721441 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ae1746-a68a-4f33-812c-b68315e0409a admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:33.721882 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-39b1e281-0337-4b7a-b2c3-cc494fa5c22d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8cac123f51ca4bd1b7ea70bd0a84396d'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:33.722207 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ae1746-a68a-4f33-812c-b68315e0409a admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:33.723082 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63ae1746-a68a-4f33-812c-b68315e0409a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:33.731695 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63ae1746-a68a-4f33-812c-b68315e0409a admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:33.739760 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-39b1e281-0337-4b7a-b2c3-cc494fa5c22d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8cac123f51ca4bd1b7ea70bd0a84396d'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:33.743050 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-39b1e281-0337-4b7a-b2c3-cc494fa5c22d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8cac123f51ca4bd1b7ea70bd0a84396d'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:33.761233 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-39b1e281-0337-4b7a-b2c3-cc494fa5c22d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8cac123f51ca4bd1b7ea70bd0a84396d'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:33.764658 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1119/2202] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:57:33 2022] GET /identity/v3/roles => generated 1420 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:57:33.773060 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63ae1746-a68a-4f33-812c-b68315e0409a admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:33.780721 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-39b1e281-0337-4b7a-b2c3-cc494fa5c22d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8cac123f51ca4bd1b7ea70bd0a84396d'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:33.787741 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-63ae1746-a68a-4f33-812c-b68315e0409a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:33.795444 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1084/2203] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 00:57:33 2022] DELETE /identity/v3/users/8cac123f51ca4bd1b7ea70bd0a84396d => generated 0 bytes in 314 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:33.801354 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65313def-cc0b-426b-a698-98c62eb3077a None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:33.801354 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65313def-cc0b-426b-a698-98c62eb3077a None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:33.801354 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65313def-cc0b-426b-a698-98c62eb3077a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:33.804655 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63ae1746-a68a-4f33-812c-b68315e0409a admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:33.841237 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-65313def-cc0b-426b-a698-98c62eb3077a None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:33.844180 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-65313def-cc0b-426b-a698-98c62eb3077a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:33.928164 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3e1b348-2226-47f7-bd79-ed7816f22673 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:33.928841 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e1b348-2226-47f7-bd79-ed7816f22673 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:33.929324 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e1b348-2226-47f7-bd79-ed7816f22673 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:33.929606 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e1b348-2226-47f7-bd79-ed7816f22673 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:33.930258 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3e1b348-2226-47f7-bd79-ed7816f22673 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:33.932274 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3e1b348-2226-47f7-bd79-ed7816f22673 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:33.948217 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1120/2204] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:57:33 2022] GET /identity/v3/roles => generated 1420 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:57:33.952793 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3e1b348-2226-47f7-bd79-ed7816f22673 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:33.968326 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1085/2205] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:57:33.971527 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3e1b348-2226-47f7-bd79-ed7816f22673 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:33.974663 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65313def-cc0b-426b-a698-98c62eb3077a None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:33.976491 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-65313def-cc0b-426b-a698-98c62eb3077a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:33.979053 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6220268-6d86-454f-b9cf-78913bf351e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:33.980095 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6220268-6d86-454f-b9cf-78913bf351e4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:33.980574 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6220268-6d86-454f-b9cf-78913bf351e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:33.981011 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6220268-6d86-454f-b9cf-78913bf351e4 admin admin] PATH_INFO: `/v3/projects/95f6d79f28744858b8cfe8c8313c2ff2/users/8acf8140969a4295a04fedd7cde5ab68/roles/588016933bab436d8b7c3ac76827356d` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:33.990161 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65313def-cc0b-426b-a698-98c62eb3077a None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:34.000085 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ad4d30a-f36d-4257-9851-4b2d6a56bb1a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:34.000814 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad4d30a-f36d-4257-9851-4b2d6a56bb1a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:34.001271 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad4d30a-f36d-4257-9851-4b2d6a56bb1a admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:34.001613 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad4d30a-f36d-4257-9851-4b2d6a56bb1a admin admin] PATH_INFO: `/v3/projects/b0adffcc6be142ee987cf24ca2c646b1` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:34.002414 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6220268-6d86-454f-b9cf-78913bf351e4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=95f6d79f28744858b8cfe8c8313c2ff2, user_id=8acf8140969a4295a04fedd7cde5ab68, role_id=588016933bab436d8b7c3ac76827356d)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:34.003318 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ad4d30a-f36d-4257-9851-4b2d6a56bb1a admin admin] RBAC: Authorizing `identity:delete_project(project_id=b0adffcc6be142ee987cf24ca2c646b1)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:34.005609 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6220268-6d86-454f-b9cf-78913bf351e4 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:34.008931 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ad4d30a-f36d-4257-9851-4b2d6a56bb1a admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:34.028675 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1121/2206] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:57:33 2022] PUT /identity/v3/projects/95f6d79f28744858b8cfe8c8313c2ff2/users/8acf8140969a4295a04fedd7cde5ab68/roles/588016933bab436d8b7c3ac76827356d => generated 0 bytes in 76 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:34.029417 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6220268-6d86-454f-b9cf-78913bf351e4 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:34.031458 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-f6220268-6d86-454f-b9cf-78913bf351e4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:34.038967 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:34.056613 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:34.119936 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-4ad4d30a-f36d-4257-9851-4b2d6a56bb1a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b0adffcc6be142ee987cf24ca2c646b1', 'request_id': 'req-4ad4d30a-f36d-4257-9851-4b2d6a56bb1a'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:34.136194 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-4ad4d30a-f36d-4257-9851-4b2d6a56bb1a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b0adffcc6be142ee987cf24ca2c646b1', 'request_id': 'req-4ad4d30a-f36d-4257-9851-4b2d6a56bb1a'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:34.173064 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-4ad4d30a-f36d-4257-9851-4b2d6a56bb1a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b0adffcc6be142ee987cf24ca2c646b1', 'request_id': 'req-4ad4d30a-f36d-4257-9851-4b2d6a56bb1a'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:34.192790 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-4ad4d30a-f36d-4257-9851-4b2d6a56bb1a admin admin] The token cache is being invalidate because project b0adffcc6be142ee987cf24ca2c646b1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81453) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 00:57:34.193117 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-4ad4d30a-f36d-4257-9851-4b2d6a56bb1a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:34.193772 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-4ad4d30a-f36d-4257-9851-4b2d6a56bb1a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:34.195279 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1086/2207] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 00:57:33 2022] DELETE /identity/v3/projects/b0adffcc6be142ee987cf24ca2c646b1 => generated 0 bytes in 221 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 09 00:57:34.196587 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d02cad06-7cd8-4a31-b424-266a69df472c None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:34.196868 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d02cad06-7cd8-4a31-b424-266a69df472c None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:34.197138 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d02cad06-7cd8-4a31-b424-266a69df472c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:34.228014 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6220268-6d86-454f-b9cf-78913bf351e4 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:34.238818 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-d02cad06-7cd8-4a31-b424-266a69df472c None None] MFA Rules not processed for user `8acf8140969a4295a04fedd7cde5ab68`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:34.240057 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-d02cad06-7cd8-4a31-b424-266a69df472c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:34.335276 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd997ac6-0419-4b00-9051-bdc178a9c81f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:34.335962 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd997ac6-0419-4b00-9051-bdc178a9c81f service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:34.336204 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd997ac6-0419-4b00-9051-bdc178a9c81f service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:34.336609 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd997ac6-0419-4b00-9051-bdc178a9c81f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:34.338543 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-fd997ac6-0419-4b00-9051-bdc178a9c81f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:34.375112 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1087/2208] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:34 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:57:34.377846 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d02cad06-7cd8-4a31-b424-266a69df472c None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:34.380366 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-d02cad06-7cd8-4a31-b424-266a69df472c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:34.432666 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd997ac6-0419-4b00-9051-bdc178a9c81f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:34.434809 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd997ac6-0419-4b00-9051-bdc178a9c81f service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:34.448773 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1122/2209] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:57:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 420 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:57:34.449601 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd997ac6-0419-4b00-9051-bdc178a9c81f service neutron] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:34.451883 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-fd997ac6-0419-4b00-9051-bdc178a9c81f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:34.470452 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd997ac6-0419-4b00-9051-bdc178a9c81f service neutron] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:34.476252 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc608162-5379-4af6-979f-a7273297be13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:34.477045 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc608162-5379-4af6-979f-a7273297be13 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:34.477335 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc608162-5379-4af6-979f-a7273297be13 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:34.477614 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc608162-5379-4af6-979f-a7273297be13 admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:34.478304 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc608162-5379-4af6-979f-a7273297be13 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:34.480472 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc608162-5379-4af6-979f-a7273297be13 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:34.495678 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d02cad06-7cd8-4a31-b424-266a69df472c None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:34.500963 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55682bde-1f2d-4809-84ec-7490d3aaf49d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:34.501859 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55682bde-1f2d-4809-84ec-7490d3aaf49d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:34.502649 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55682bde-1f2d-4809-84ec-7490d3aaf49d admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:34.502959 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55682bde-1f2d-4809-84ec-7490d3aaf49d admin admin] PATH_INFO: `/v3/users/10570712215149159e5517b255ec6637` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:34.504823 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55682bde-1f2d-4809-84ec-7490d3aaf49d admin admin] RBAC: Authorizing `identity:delete_user(user_id=10570712215149159e5517b255ec6637)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:34.507066 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55682bde-1f2d-4809-84ec-7490d3aaf49d admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:34.536613 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1123/2210] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 00:57:34 2022] POST /identity/v3/projects => generated 377 bytes in 87 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:57:34.541436 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc608162-5379-4af6-979f-a7273297be13 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:34.574731 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:34.616054 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-55682bde-1f2d-4809-84ec-7490d3aaf49d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '10570712215149159e5517b255ec6637'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:34.639044 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-55682bde-1f2d-4809-84ec-7490d3aaf49d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '10570712215149159e5517b255ec6637'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:34.643169 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-55682bde-1f2d-4809-84ec-7490d3aaf49d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '10570712215149159e5517b255ec6637'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:34.652066 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:34.662165 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-55682bde-1f2d-4809-84ec-7490d3aaf49d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '10570712215149159e5517b255ec6637'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:34.675909 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc608162-5379-4af6-979f-a7273297be13 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:34.682192 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-55682bde-1f2d-4809-84ec-7490d3aaf49d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '10570712215149159e5517b255ec6637'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:34.692901 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1088/2211] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 00:57:34 2022] DELETE /identity/v3/users/10570712215149159e5517b255ec6637 => generated 0 bytes in 317 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:34.695820 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b07e0d1e-570e-4969-9740-5c300d54c1c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:34.695820 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b07e0d1e-570e-4969-9740-5c300d54c1c7 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:34.695820 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b07e0d1e-570e-4969-9740-5c300d54c1c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:34.695820 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b07e0d1e-570e-4969-9740-5c300d54c1c7 admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:34.696377 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b07e0d1e-570e-4969-9740-5c300d54c1c7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:34.700356 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b07e0d1e-570e-4969-9740-5c300d54c1c7 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:34.771666 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1124/2212] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:57:34 2022] POST /identity/v3/users => generated 295 bytes in 230 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:57:34.775533 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55682bde-1f2d-4809-84ec-7490d3aaf49d admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:34.777029 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-55682bde-1f2d-4809-84ec-7490d3aaf49d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:34.923647 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55682bde-1f2d-4809-84ec-7490d3aaf49d admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:34.940178 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92812fab-cc0e-4e0a-b1fe-364f5d4aab81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:34.941584 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92812fab-cc0e-4e0a-b1fe-364f5d4aab81 admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:34.942131 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92812fab-cc0e-4e0a-b1fe-364f5d4aab81 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:34.942482 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92812fab-cc0e-4e0a-b1fe-364f5d4aab81 admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:34.943205 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92812fab-cc0e-4e0a-b1fe-364f5d4aab81 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:34.945713 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92812fab-cc0e-4e0a-b1fe-364f5d4aab81 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:34.960425 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1089/2213] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:57:34 2022] GET /identity/v3/roles => generated 1420 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:57:34.964750 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b07e0d1e-570e-4969-9740-5c300d54c1c7 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:34.981391 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b07e0d1e-570e-4969-9740-5c300d54c1c7 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:34.988819 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a7cc737-9b04-477b-931e-dd590ab44e74 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:34.989830 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a7cc737-9b04-477b-931e-dd590ab44e74 admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:34.990448 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a7cc737-9b04-477b-931e-dd590ab44e74 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:34.990956 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a7cc737-9b04-477b-931e-dd590ab44e74 admin admin] PATH_INFO: `/v3/projects/a242229ba2ba4d10b765848a9cdd89a5/users/e8f4849ef4bd428b9489acabdda54b95/roles/092036ee46ac427baf4d56468575f697` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:35.013995 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a7cc737-9b04-477b-931e-dd590ab44e74 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a242229ba2ba4d10b765848a9cdd89a5, user_id=e8f4849ef4bd428b9489acabdda54b95, role_id=092036ee46ac427baf4d56468575f697)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:35.017916 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a7cc737-9b04-477b-931e-dd590ab44e74 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:35.063296 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1125/2214] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:57:34 2022] PUT /identity/v3/projects/a242229ba2ba4d10b765848a9cdd89a5/users/e8f4849ef4bd428b9489acabdda54b95/roles/092036ee46ac427baf4d56468575f697 => generated 0 bytes in 96 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:35.064872 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92812fab-cc0e-4e0a-b1fe-364f5d4aab81 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:35.094616 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92812fab-cc0e-4e0a-b1fe-364f5d4aab81 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:35.208204 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e04fdce8-a041-4911-ae4b-4b025ee74746 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:35.208970 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e04fdce8-a041-4911-ae4b-4b025ee74746 admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:35.210011 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e04fdce8-a041-4911-ae4b-4b025ee74746 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:35.210665 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e04fdce8-a041-4911-ae4b-4b025ee74746 admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:35.211708 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e04fdce8-a041-4911-ae4b-4b025ee74746 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:35.213875 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e04fdce8-a041-4911-ae4b-4b025ee74746 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:35.228966 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1090/2215] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:57:35 2022] GET /identity/v3/roles => generated 1420 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:57:35.233338 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a7cc737-9b04-477b-931e-dd590ab44e74 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:35.245479 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a7cc737-9b04-477b-931e-dd590ab44e74 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:35.250312 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23ed1b3d-0e93-4f85-8b10-662f88080035 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:35.250969 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23ed1b3d-0e93-4f85-8b10-662f88080035 admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:35.251513 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23ed1b3d-0e93-4f85-8b10-662f88080035 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:35.251659 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23ed1b3d-0e93-4f85-8b10-662f88080035 admin admin] PATH_INFO: `/v3/projects/a242229ba2ba4d10b765848a9cdd89a5/users/e8f4849ef4bd428b9489acabdda54b95/roles/092036ee46ac427baf4d56468575f697` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:35.254054 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23ed1b3d-0e93-4f85-8b10-662f88080035 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a242229ba2ba4d10b765848a9cdd89a5, user_id=e8f4849ef4bd428b9489acabdda54b95, role_id=092036ee46ac427baf4d56468575f697)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:35.259242 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23ed1b3d-0e93-4f85-8b10-662f88080035 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:35.260355 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e04fdce8-a041-4911-ae4b-4b025ee74746 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:35.262159 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-e04fdce8-a041-4911-ae4b-4b025ee74746 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:35.289693 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e04fdce8-a041-4911-ae4b-4b025ee74746 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:35.293483 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1126/2216] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:57:35 2022] PUT /identity/v3/projects/a242229ba2ba4d10b765848a9cdd89a5/users/e8f4849ef4bd428b9489acabdda54b95/roles/092036ee46ac427baf4d56468575f697 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 09 00:57:35.307115 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1122ae8-70f0-4a87-8361-ef9cb26dc65a None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:35.307863 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1122ae8-70f0-4a87-8361-ef9cb26dc65a None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:35.308214 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1122ae8-70f0-4a87-8361-ef9cb26dc65a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:35.395700 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-c1122ae8-70f0-4a87-8361-ef9cb26dc65a None None] MFA Rules not processed for user `e8f4849ef4bd428b9489acabdda54b95`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:35.397623 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-c1122ae8-70f0-4a87-8361-ef9cb26dc65a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:35.420259 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:35.462030 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-634bb274-a895-48a7-b196-fb1e2503099f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:35.463636 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634bb274-a895-48a7-b196-fb1e2503099f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:35.466127 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634bb274-a895-48a7-b196-fb1e2503099f admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:35.466503 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634bb274-a895-48a7-b196-fb1e2503099f admin admin] PATH_INFO: `/v3/projects/0571ef472ec449f5bf821d4c27e43c90` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:35.471704 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-634bb274-a895-48a7-b196-fb1e2503099f admin admin] RBAC: Authorizing `identity:delete_project(project_id=0571ef472ec449f5bf821d4c27e43c90)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:35.475059 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-634bb274-a895-48a7-b196-fb1e2503099f admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:35.581970 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1127/2217] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:35 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 277 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:57:35.584739 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1122ae8-70f0-4a87-8361-ef9cb26dc65a None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:35.586293 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-c1122ae8-70f0-4a87-8361-ef9cb26dc65a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:35.591052 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-634bb274-a895-48a7-b196-fb1e2503099f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0571ef472ec449f5bf821d4c27e43c90', 'request_id': 'req-634bb274-a895-48a7-b196-fb1e2503099f'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:35.599152 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-634bb274-a895-48a7-b196-fb1e2503099f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0571ef472ec449f5bf821d4c27e43c90', 'request_id': 'req-634bb274-a895-48a7-b196-fb1e2503099f'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:35.609618 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-634bb274-a895-48a7-b196-fb1e2503099f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0571ef472ec449f5bf821d4c27e43c90', 'request_id': 'req-634bb274-a895-48a7-b196-fb1e2503099f'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:35.620899 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1122ae8-70f0-4a87-8361-ef9cb26dc65a None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:35.638999 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-634bb274-a895-48a7-b196-fb1e2503099f admin admin] The token cache is being invalidate because project 0571ef472ec449f5bf821d4c27e43c90 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81453) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 00:57:35.639557 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-634bb274-a895-48a7-b196-fb1e2503099f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:35.640940 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-634bb274-a895-48a7-b196-fb1e2503099f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:35.652965 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1091/2218] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 00:57:35 2022] DELETE /identity/v3/projects/0571ef472ec449f5bf821d4c27e43c90 => generated 0 bytes in 393 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 09 00:57:35.657276 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-634bb274-a895-48a7-b196-fb1e2503099f admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:35.658893 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-634bb274-a895-48a7-b196-fb1e2503099f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:35.721636 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e0cc9ec-0503-4847-9348-da353023a1bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:35.723322 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0cc9ec-0503-4847-9348-da353023a1bf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:35.723741 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0cc9ec-0503-4847-9348-da353023a1bf admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:35.724252 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0cc9ec-0503-4847-9348-da353023a1bf admin admin] PATH_INFO: `/v3/projects/f5a5a1e0aa9d43fba6e79cb7d035241f` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:35.725748 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e0cc9ec-0503-4847-9348-da353023a1bf admin admin] RBAC: Authorizing `identity:delete_project(project_id=f5a5a1e0aa9d43fba6e79cb7d035241f)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:35.727865 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e0cc9ec-0503-4847-9348-da353023a1bf admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:35.768911 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-634bb274-a895-48a7-b196-fb1e2503099f admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:35.782663 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9d99524-88aa-4f8d-94a6-c23948cdbae8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3f48335668a4e8f934ac11ff66cd308', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:35.783560 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d99524-88aa-4f8d-94a6-c23948cdbae8 service cinder] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:35.784092 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=93,keystone:INSERT=5,keystone:DELETE=2 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:35.786108 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d99524-88aa-4f8d-94a6-c23948cdbae8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:35.787345 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d99524-88aa-4f8d-94a6-c23948cdbae8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:35.797027 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-e9d99524-88aa-4f8d-94a6-c23948cdbae8 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:35.820204 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-1e0cc9ec-0503-4847-9348-da353023a1bf admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f5a5a1e0aa9d43fba6e79cb7d035241f', 'request_id': 'req-1e0cc9ec-0503-4847-9348-da353023a1bf'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:35.836938 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-1e0cc9ec-0503-4847-9348-da353023a1bf admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f5a5a1e0aa9d43fba6e79cb7d035241f', 'request_id': 'req-1e0cc9ec-0503-4847-9348-da353023a1bf'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:35.862971 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-1e0cc9ec-0503-4847-9348-da353023a1bf admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f5a5a1e0aa9d43fba6e79cb7d035241f', 'request_id': 'req-1e0cc9ec-0503-4847-9348-da353023a1bf'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:35.885515 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-1e0cc9ec-0503-4847-9348-da353023a1bf admin admin] The token cache is being invalidate because project f5a5a1e0aa9d43fba6e79cb7d035241f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 00:57:35.885867 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-1e0cc9ec-0503-4847-9348-da353023a1bf admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:35.886543 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-1e0cc9ec-0503-4847-9348-da353023a1bf admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:35.890610 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1128/2219] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 00:57:35 2022] DELETE /identity/v3/projects/f5a5a1e0aa9d43fba6e79cb7d035241f => generated 0 bytes in 308 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:35.891279 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e0cc9ec-0503-4847-9348-da353023a1bf admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:35.892712 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-1e0cc9ec-0503-4847-9348-da353023a1bf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:35.948894 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9d99524-88aa-4f8d-94a6-c23948cdbae8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:35.950525 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9d99524-88aa-4f8d-94a6-c23948cdbae8 service cinder] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:35.963125 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1092/2220] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:57:35 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 306 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:57:35.964390 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e066888-e64f-4cfb-8ef8-775fa04c9e5f None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:35.964677 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e066888-e64f-4cfb-8ef8-775fa04c9e5f None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:35.965030 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e066888-e64f-4cfb-8ef8-775fa04c9e5f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:35.990051 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-5e066888-e64f-4cfb-8ef8-775fa04c9e5f None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:35.991399 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-5e066888-e64f-4cfb-8ef8-775fa04c9e5f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:36.016123 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e0cc9ec-0503-4847-9348-da353023a1bf admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:36.023971 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-181dda49-822e-4df5-b5ea-9293bdfad216 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:36.024802 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-181dda49-822e-4df5-b5ea-9293bdfad216 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:36.025262 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-181dda49-822e-4df5-b5ea-9293bdfad216 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:36.025509 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-181dda49-822e-4df5-b5ea-9293bdfad216 admin admin] PATH_INFO: `/v3/users/3430bd39961e487a89a9ba08aea8a43d` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:36.026741 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-181dda49-822e-4df5-b5ea-9293bdfad216 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3430bd39961e487a89a9ba08aea8a43d)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:36.028789 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-181dda49-822e-4df5-b5ea-9293bdfad216 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:36.034484 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:36.154748 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-181dda49-822e-4df5-b5ea-9293bdfad216 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3430bd39961e487a89a9ba08aea8a43d'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:36.168219 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-181dda49-822e-4df5-b5ea-9293bdfad216 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3430bd39961e487a89a9ba08aea8a43d'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:36.170319 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-181dda49-822e-4df5-b5ea-9293bdfad216 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3430bd39961e487a89a9ba08aea8a43d'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:36.170855 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1093/2221] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 207 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:57:36.171774 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e066888-e64f-4cfb-8ef8-775fa04c9e5f None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:36.173247 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-5e066888-e64f-4cfb-8ef8-775fa04c9e5f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:36.176995 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-181dda49-822e-4df5-b5ea-9293bdfad216 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3430bd39961e487a89a9ba08aea8a43d'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:36.190150 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-181dda49-822e-4df5-b5ea-9293bdfad216 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3430bd39961e487a89a9ba08aea8a43d'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:36.220880 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1129/2222] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 00:57:35 2022] DELETE /identity/v3/users/3430bd39961e487a89a9ba08aea8a43d => generated 0 bytes in 330 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:36.222585 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ada87ae-ff92-47d7-9635-6153a474e5da None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:36.223035 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ada87ae-ff92-47d7-9635-6153a474e5da None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:36.223723 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ada87ae-ff92-47d7-9635-6153a474e5da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:36.261949 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-6ada87ae-ff92-47d7-9635-6153a474e5da None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:36.263176 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-6ada87ae-ff92-47d7-9635-6153a474e5da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:36.266991 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e066888-e64f-4cfb-8ef8-775fa04c9e5f None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:36.320441 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=83,keystone:DELETE=15,keystone:INSERT=2 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:36.365462 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1130/2223] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:57:36.369932 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ada87ae-ff92-47d7-9635-6153a474e5da None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:36.380697 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ada87ae-ff92-47d7-9635-6153a474e5da None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:36.385831 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9026bd1-5d28-4223-8a72-72a8855fad18 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:36.386670 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9026bd1-5d28-4223-8a72-72a8855fad18 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:36.386978 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9026bd1-5d28-4223-8a72-72a8855fad18 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:36.387271 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9026bd1-5d28-4223-8a72-72a8855fad18 admin admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:36.387925 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9026bd1-5d28-4223-8a72-72a8855fad18 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:57:36.388354 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9026bd1-5d28-4223-8a72-72a8855fad18 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:36.388715 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac984204-bafe-452b-96f7-2e20b3270988 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3f48335668a4e8f934ac11ff66cd308', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:36.389792 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac984204-bafe-452b-96f7-2e20b3270988 service cinder] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:36.390240 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9026bd1-5d28-4223-8a72-72a8855fad18 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:36.390451 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac984204-bafe-452b-96f7-2e20b3270988 service cinder] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:36.390858 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac984204-bafe-452b-96f7-2e20b3270988 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:36.394740 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-ac984204-bafe-452b-96f7-2e20b3270988 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:36.409915 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1131/2224] 10.222.0.33 () {58 vars in 1153 bytes} [Sun Oct 9 00:57:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:57:36.411235 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9026bd1-5d28-4223-8a72-72a8855fad18 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:36.423310 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9026bd1-5d28-4223-8a72-72a8855fad18 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:36.431082 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f03fabed-a2a7-4ad6-ac67-08a0bd199b7d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:36.431716 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f03fabed-a2a7-4ad6-ac67-08a0bd199b7d admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:36.432009 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f03fabed-a2a7-4ad6-ac67-08a0bd199b7d admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:36.432250 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f03fabed-a2a7-4ad6-ac67-08a0bd199b7d admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:36.433012 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f03fabed-a2a7-4ad6-ac67-08a0bd199b7d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:36.435186 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f03fabed-a2a7-4ad6-ac67-08a0bd199b7d admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:36.481023 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1132/2225] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 00:57:36 2022] POST /identity/v3/projects => generated 363 bytes in 70 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:57:36.486102 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f03fabed-a2a7-4ad6-ac67-08a0bd199b7d admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:36.493882 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac984204-bafe-452b-96f7-2e20b3270988 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:36.495117 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac984204-bafe-452b-96f7-2e20b3270988 service cinder] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:36.505967 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1094/2226] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:57:36 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 334 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:57:36.595237 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f03fabed-a2a7-4ad6-ac67-08a0bd199b7d admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:36.610341 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f69a95d1-77f3-43b4-80a1-5f91ea93c0e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:36.611422 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f69a95d1-77f3-43b4-80a1-5f91ea93c0e8 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:36.611422 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f69a95d1-77f3-43b4-80a1-5f91ea93c0e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:36.612306 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f69a95d1-77f3-43b4-80a1-5f91ea93c0e8 admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:36.615386 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f69a95d1-77f3-43b4-80a1-5f91ea93c0e8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:36.616353 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f69a95d1-77f3-43b4-80a1-5f91ea93c0e8 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:36.667556 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1133/2227] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:57:36 2022] POST /identity/v3/users => generated 282 bytes in 182 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:57:36.668148 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac984204-bafe-452b-96f7-2e20b3270988 service cinder] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:36.681252 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac984204-bafe-452b-96f7-2e20b3270988 service cinder] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:36.695366 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b551daa0-1aaf-418b-93e3-94a49f502612 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:36.696924 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b551daa0-1aaf-418b-93e3-94a49f502612 admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:36.696924 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b551daa0-1aaf-418b-93e3-94a49f502612 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:36.696924 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b551daa0-1aaf-418b-93e3-94a49f502612 admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:36.699562 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b551daa0-1aaf-418b-93e3-94a49f502612 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:36.705642 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b551daa0-1aaf-418b-93e3-94a49f502612 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:36.706811 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f69a95d1-77f3-43b4-80a1-5f91ea93c0e8 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:36.708947 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-f69a95d1-77f3-43b4-80a1-5f91ea93c0e8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:36.730555 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f69a95d1-77f3-43b4-80a1-5f91ea93c0e8 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:36.739731 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1095/2228] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:57:36 2022] GET /identity/v3/roles => generated 1420 bytes in 72 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 09 00:57:36.742525 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b551daa0-1aaf-418b-93e3-94a49f502612 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:36.746900 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d947604-c44c-4cce-8fc9-0c51abefbbd1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:36.747550 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d947604-c44c-4cce-8fc9-0c51abefbbd1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:36.747863 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d947604-c44c-4cce-8fc9-0c51abefbbd1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:36.748137 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d947604-c44c-4cce-8fc9-0c51abefbbd1 admin admin] PATH_INFO: `/v3/projects/8ddaf92644c048f39d70583218cec933` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:36.751846 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d947604-c44c-4cce-8fc9-0c51abefbbd1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8ddaf92644c048f39d70583218cec933)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:36.754485 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d947604-c44c-4cce-8fc9-0c51abefbbd1 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:36.763626 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b551daa0-1aaf-418b-93e3-94a49f502612 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:36.781242 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08c73a70-60c0-4f1b-8291-f63c202c7e45 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:36.782314 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c73a70-60c0-4f1b-8291-f63c202c7e45 admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:36.782882 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c73a70-60c0-4f1b-8291-f63c202c7e45 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:36.783249 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c73a70-60c0-4f1b-8291-f63c202c7e45 admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:36.785322 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08c73a70-60c0-4f1b-8291-f63c202c7e45 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:36.789472 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08c73a70-60c0-4f1b-8291-f63c202c7e45 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:36.811854 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1096/2229] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:57:36 2022] GET /identity/v3/roles => generated 1420 bytes in 70 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:57:36.816228 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08c73a70-60c0-4f1b-8291-f63c202c7e45 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:36.848066 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08c73a70-60c0-4f1b-8291-f63c202c7e45 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:36.858826 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:36.863728 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-0d947604-c44c-4cce-8fc9-0c51abefbbd1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8ddaf92644c048f39d70583218cec933', 'request_id': 'req-0d947604-c44c-4cce-8fc9-0c51abefbbd1'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:36.873769 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-0d947604-c44c-4cce-8fc9-0c51abefbbd1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8ddaf92644c048f39d70583218cec933', 'request_id': 'req-0d947604-c44c-4cce-8fc9-0c51abefbbd1'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:36.892193 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-0d947604-c44c-4cce-8fc9-0c51abefbbd1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8ddaf92644c048f39d70583218cec933', 'request_id': 'req-0d947604-c44c-4cce-8fc9-0c51abefbbd1'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:36.918072 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-0d947604-c44c-4cce-8fc9-0c51abefbbd1 admin admin] The token cache is being invalidate because project 8ddaf92644c048f39d70583218cec933 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 00:57:36.918374 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-0d947604-c44c-4cce-8fc9-0c51abefbbd1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:36.919157 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-0d947604-c44c-4cce-8fc9-0c51abefbbd1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:36.923791 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1134/2230] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 00:57:36 2022] DELETE /identity/v3/projects/8ddaf92644c048f39d70583218cec933 => generated 0 bytes in 218 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 09 00:57:36.928335 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d947604-c44c-4cce-8fc9-0c51abefbbd1 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:36.931518 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-0d947604-c44c-4cce-8fc9-0c51abefbbd1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:36.982455 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=88,keystone:INSERT=5,keystone:DELETE=7 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:36.999235 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adc2205b-23c4-411c-bba9-57381b0a77c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:37.000080 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc2205b-23c4-411c-bba9-57381b0a77c4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:37.000310 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc2205b-23c4-411c-bba9-57381b0a77c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:37.000643 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc2205b-23c4-411c-bba9-57381b0a77c4 admin admin] PATH_INFO: `/v3/projects/ebbbad64bcf24f6c9667d7a5282a0f00/users/b5ee382f8e4642589c39c6f49d7615d2/roles/588016933bab436d8b7c3ac76827356d` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:37.023352 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adc2205b-23c4-411c-bba9-57381b0a77c4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ebbbad64bcf24f6c9667d7a5282a0f00, user_id=b5ee382f8e4642589c39c6f49d7615d2, role_id=588016933bab436d8b7c3ac76827356d)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:37.027666 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adc2205b-23c4-411c-bba9-57381b0a77c4 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:37.047039 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1097/2231] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:57:36 2022] PUT /identity/v3/projects/ebbbad64bcf24f6c9667d7a5282a0f00/users/b5ee382f8e4642589c39c6f49d7615d2/roles/588016933bab436d8b7c3ac76827356d => generated 0 bytes in 231 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:37.059584 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2acd97e9-6285-46cf-a139-55e23a42338c None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:37.060103 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2acd97e9-6285-46cf-a139-55e23a42338c None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:37.060447 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2acd97e9-6285-46cf-a139-55e23a42338c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:37.074620 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d947604-c44c-4cce-8fc9-0c51abefbbd1 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:37.111573 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-2acd97e9-6285-46cf-a139-55e23a42338c None None] MFA Rules not processed for user `b5ee382f8e4642589c39c6f49d7615d2`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:37.113038 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-2acd97e9-6285-46cf-a139-55e23a42338c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:37.150683 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ce9b551-6b57-41b0-8f8d-bb753e0fc5e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:37.151512 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce9b551-6b57-41b0-8f8d-bb753e0fc5e0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:37.151830 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce9b551-6b57-41b0-8f8d-bb753e0fc5e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:37.152106 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce9b551-6b57-41b0-8f8d-bb753e0fc5e0 admin admin] PATH_INFO: `/v3/users/3e6e6d692fe943c188f077ae75ce3062` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:37.153374 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ce9b551-6b57-41b0-8f8d-bb753e0fc5e0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3e6e6d692fe943c188f077ae75ce3062)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:37.165636 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ce9b551-6b57-41b0-8f8d-bb753e0fc5e0 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:37.248588 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-6ce9b551-6b57-41b0-8f8d-bb753e0fc5e0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3e6e6d692fe943c188f077ae75ce3062'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:37.265328 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-6ce9b551-6b57-41b0-8f8d-bb753e0fc5e0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3e6e6d692fe943c188f077ae75ce3062'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:37.265328 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-6ce9b551-6b57-41b0-8f8d-bb753e0fc5e0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3e6e6d692fe943c188f077ae75ce3062'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:37.265328 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-6ce9b551-6b57-41b0-8f8d-bb753e0fc5e0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3e6e6d692fe943c188f077ae75ce3062'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:37.266290 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1098/2232] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:37 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 208 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 09 00:57:37.274912 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2acd97e9-6285-46cf-a139-55e23a42338c None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:37.278598 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-2acd97e9-6285-46cf-a139-55e23a42338c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:37.282893 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-6ce9b551-6b57-41b0-8f8d-bb753e0fc5e0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3e6e6d692fe943c188f077ae75ce3062'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:37.317076 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1135/2233] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 00:57:36 2022] DELETE /identity/v3/users/3e6e6d692fe943c188f077ae75ce3062 => generated 0 bytes in 389 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:37.364701 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2acd97e9-6285-46cf-a139-55e23a42338c None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:37.432044 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:37.476640 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96190227-2d61-4989-a37b-2c238d773566 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3f48335668a4e8f934ac11ff66cd308', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:37.477262 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96190227-2d61-4989-a37b-2c238d773566 service cinder] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:37.477541 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96190227-2d61-4989-a37b-2c238d773566 service cinder] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:37.477811 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96190227-2d61-4989-a37b-2c238d773566 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:37.479621 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-96190227-2d61-4989-a37b-2c238d773566 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:37.565713 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96190227-2d61-4989-a37b-2c238d773566 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:37.567661 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96190227-2d61-4989-a37b-2c238d773566 service cinder] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:37.577744 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1099/2234] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:57:37 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 304 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:57:37.821503 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ce9b551-6b57-41b0-8f8d-bb753e0fc5e0 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:37.823795 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-6ce9b551-6b57-41b0-8f8d-bb753e0fc5e0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:38.029136 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:38.071619 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ce9b551-6b57-41b0-8f8d-bb753e0fc5e0 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:38.091556 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eeb2323-1c5a-49a7-9db7-1d3cf83f4445 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:38.093541 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eeb2323-1c5a-49a7-9db7-1d3cf83f4445 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:38.093828 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eeb2323-1c5a-49a7-9db7-1d3cf83f4445 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:38.094104 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eeb2323-1c5a-49a7-9db7-1d3cf83f4445 admin admin] PATH_INFO: `/v3/projects/b03af7cb152a4c6ab87b38c24e1fe7af` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:38.098293 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eeb2323-1c5a-49a7-9db7-1d3cf83f4445 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b03af7cb152a4c6ab87b38c24e1fe7af)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:38.109732 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eeb2323-1c5a-49a7-9db7-1d3cf83f4445 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:38.125360 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ebbf78-e1b3-4be1-9eeb-4eabdd070b97 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:38.126085 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ebbf78-e1b3-4be1-9eeb-4eabdd070b97 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:38.126085 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ebbf78-e1b3-4be1-9eeb-4eabdd070b97 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:38.167658 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-f1ebbf78-e1b3-4be1-9eeb-4eabdd070b97 None None] MFA Rules not processed for user `04bc22d558c2466ba43a456d08bdd6bd`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:38.169130 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-f1ebbf78-e1b3-4be1-9eeb-4eabdd070b97 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:38.240483 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-5eeb2323-1c5a-49a7-9db7-1d3cf83f4445 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b03af7cb152a4c6ab87b38c24e1fe7af', 'request_id': 'req-5eeb2323-1c5a-49a7-9db7-1d3cf83f4445'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:38.250514 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-5eeb2323-1c5a-49a7-9db7-1d3cf83f4445 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b03af7cb152a4c6ab87b38c24e1fe7af', 'request_id': 'req-5eeb2323-1c5a-49a7-9db7-1d3cf83f4445'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:38.261769 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-5eeb2323-1c5a-49a7-9db7-1d3cf83f4445 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b03af7cb152a4c6ab87b38c24e1fe7af', 'request_id': 'req-5eeb2323-1c5a-49a7-9db7-1d3cf83f4445'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:38.284322 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1100/2235] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:38 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 09 00:57:38.295749 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1ebbf78-e1b3-4be1-9eeb-4eabdd070b97 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:38.295749 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-f1ebbf78-e1b3-4be1-9eeb-4eabdd070b97 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:38.307872 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-5eeb2323-1c5a-49a7-9db7-1d3cf83f4445 admin admin] The token cache is being invalidate because project b03af7cb152a4c6ab87b38c24e1fe7af was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 00:57:38.308168 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-5eeb2323-1c5a-49a7-9db7-1d3cf83f4445 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:38.309123 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-5eeb2323-1c5a-49a7-9db7-1d3cf83f4445 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:38.310965 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1136/2236] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 00:57:37 2022] DELETE /identity/v3/projects/b03af7cb152a4c6ab87b38c24e1fe7af => generated 0 bytes in 490 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:38.315875 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45458687-9559-4036-a178-addbbc0c77ee None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:38.316590 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1ebbf78-e1b3-4be1-9eeb-4eabdd070b97 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:38.316841 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45458687-9559-4036-a178-addbbc0c77ee None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:38.317187 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45458687-9559-4036-a178-addbbc0c77ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:38.340687 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-45458687-9559-4036-a178-addbbc0c77ee None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:38.342020 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-45458687-9559-4036-a178-addbbc0c77ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:38.390790 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e73a8905-d9ac-472c-bbdc-3be8fc7fcaca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:38.391425 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e73a8905-d9ac-472c-bbdc-3be8fc7fcaca admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:38.391745 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e73a8905-d9ac-472c-bbdc-3be8fc7fcaca admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:38.392020 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e73a8905-d9ac-472c-bbdc-3be8fc7fcaca admin admin] PATH_INFO: `/v3/projects` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:38.392681 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e73a8905-d9ac-472c-bbdc-3be8fc7fcaca admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:38.396846 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e73a8905-d9ac-472c-bbdc-3be8fc7fcaca admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:38.424162 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1137/2237] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:57:38.430701 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1101/2238] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 00:57:38 2022] POST /identity/v3/projects => generated 377 bytes in 142 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:57:38.432852 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6bf3f07-4451-4e2f-86b7-ba9dd05c8a9e None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:38.433258 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6bf3f07-4451-4e2f-86b7-ba9dd05c8a9e None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:38.433732 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6bf3f07-4451-4e2f-86b7-ba9dd05c8a9e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:38.434898 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e73a8905-d9ac-472c-bbdc-3be8fc7fcaca admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:38.436625 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-e73a8905-d9ac-472c-bbdc-3be8fc7fcaca admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:38.449939 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:38.457510 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-b6bf3f07-4451-4e2f-86b7-ba9dd05c8a9e None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:38.458832 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-b6bf3f07-4451-4e2f-86b7-ba9dd05c8a9e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:38.511397 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e73a8905-d9ac-472c-bbdc-3be8fc7fcaca admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:38.518230 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd332806-1396-4b90-a955-24b74f543ce6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:38.519009 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd332806-1396-4b90-a955-24b74f543ce6 admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:38.519708 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd332806-1396-4b90-a955-24b74f543ce6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:38.519997 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd332806-1396-4b90-a955-24b74f543ce6 admin admin] PATH_INFO: `/v3/users` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:38.520798 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd332806-1396-4b90-a955-24b74f543ce6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:38.523327 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd332806-1396-4b90-a955-24b74f543ce6 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:38.536265 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1138/2239] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:57:38.541151 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6bf3f07-4451-4e2f-86b7-ba9dd05c8a9e None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:38.549717 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1102/2240] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:57:38 2022] POST /identity/v3/users => generated 295 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:57:38.550774 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6bf3f07-4451-4e2f-86b7-ba9dd05c8a9e None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:38.553639 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd332806-1396-4b90-a955-24b74f543ce6 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:38.555457 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa92f8a1-081f-43f8-b4a0-cef80071c614 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:38.556102 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa92f8a1-081f-43f8-b4a0-cef80071c614 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:38.556357 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa92f8a1-081f-43f8-b4a0-cef80071c614 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:38.556630 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa92f8a1-081f-43f8-b4a0-cef80071c614 admin admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:38.557274 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa92f8a1-081f-43f8-b4a0-cef80071c614 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:57:38.557574 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa92f8a1-081f-43f8-b4a0-cef80071c614 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:38.559164 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa92f8a1-081f-43f8-b4a0-cef80071c614 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:38.565851 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd332806-1396-4b90-a955-24b74f543ce6 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:38.571562 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccc865e2-aba4-4dc6-b392-1ce7df110548 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:38.572279 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccc865e2-aba4-4dc6-b392-1ce7df110548 admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:38.572565 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccc865e2-aba4-4dc6-b392-1ce7df110548 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:38.572710 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1139/2241] 10.222.0.33 () {58 vars in 1153 bytes} [Sun Oct 9 00:57:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:57:38.572925 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccc865e2-aba4-4dc6-b392-1ce7df110548 admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:38.573644 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccc865e2-aba4-4dc6-b392-1ce7df110548 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:38.575321 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccc865e2-aba4-4dc6-b392-1ce7df110548 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:38.576823 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa92f8a1-081f-43f8-b4a0-cef80071c614 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:38.583058 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:38.591072 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1103/2242] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:57:38 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:57:38.591253 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa92f8a1-081f-43f8-b4a0-cef80071c614 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:38.592446 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccc865e2-aba4-4dc6-b392-1ce7df110548 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:38.597943 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25f5fac4-fd72-42f9-83bb-05720ecf3793 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:38.598982 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25f5fac4-fd72-42f9-83bb-05720ecf3793 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:38.599359 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25f5fac4-fd72-42f9-83bb-05720ecf3793 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:38.599773 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25f5fac4-fd72-42f9-83bb-05720ecf3793 admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:38.600551 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25f5fac4-fd72-42f9-83bb-05720ecf3793 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:38.602568 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25f5fac4-fd72-42f9-83bb-05720ecf3793 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:38.602965 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccc865e2-aba4-4dc6-b392-1ce7df110548 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:38.607875 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a0b5b14-999f-4ae9-a0eb-5d7d1ccf6ca3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:38.608539 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a0b5b14-999f-4ae9-a0eb-5d7d1ccf6ca3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:38.608820 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a0b5b14-999f-4ae9-a0eb-5d7d1ccf6ca3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:38.609102 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a0b5b14-999f-4ae9-a0eb-5d7d1ccf6ca3 admin admin] PATH_INFO: `/v3/projects/cac43224e7954c44a71839707ffcde6c/users/88f506c18af44d3e8ea2f64bd627d804/roles/092036ee46ac427baf4d56468575f697` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:38.626497 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a0b5b14-999f-4ae9-a0eb-5d7d1ccf6ca3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=cac43224e7954c44a71839707ffcde6c, user_id=88f506c18af44d3e8ea2f64bd627d804, role_id=092036ee46ac427baf4d56468575f697)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:38.630073 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a0b5b14-999f-4ae9-a0eb-5d7d1ccf6ca3 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:38.636551 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1140/2243] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 00:57:38 2022] POST /identity/v3/projects => generated 371 bytes in 60 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 09 00:57:38.641386 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25f5fac4-fd72-42f9-83bb-05720ecf3793 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:38.643553 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1104/2244] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:57:38 2022] PUT /identity/v3/projects/cac43224e7954c44a71839707ffcde6c/users/88f506c18af44d3e8ea2f64bd627d804/roles/092036ee46ac427baf4d56468575f697 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 09 00:57:38.647525 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a0b5b14-999f-4ae9-a0eb-5d7d1ccf6ca3 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:38.717714 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25f5fac4-fd72-42f9-83bb-05720ecf3793 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:38.724731 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a09b827-b7ff-459d-86c9-0ed7abed6d4b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:38.725463 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a09b827-b7ff-459d-86c9-0ed7abed6d4b admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:38.725801 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a09b827-b7ff-459d-86c9-0ed7abed6d4b admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:38.726137 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a09b827-b7ff-459d-86c9-0ed7abed6d4b admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:38.726418 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a0b5b14-999f-4ae9-a0eb-5d7d1ccf6ca3 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:38.726907 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a09b827-b7ff-459d-86c9-0ed7abed6d4b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:38.729009 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a09b827-b7ff-459d-86c9-0ed7abed6d4b admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:38.737245 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3155fab1-900e-4694-9edb-caac45182f31 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:38.738858 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3155fab1-900e-4694-9edb-caac45182f31 admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:38.739210 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3155fab1-900e-4694-9edb-caac45182f31 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:38.740154 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3155fab1-900e-4694-9edb-caac45182f31 admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:38.751973 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3155fab1-900e-4694-9edb-caac45182f31 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:38.753136 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1141/2245] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:57:38 2022] POST /identity/v3/users => generated 286 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:57:38.755634 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3155fab1-900e-4694-9edb-caac45182f31 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:38.757654 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a09b827-b7ff-459d-86c9-0ed7abed6d4b admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:38.771621 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1105/2246] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:57:38 2022] GET /identity/v3/roles => generated 1420 bytes in 123 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 09 00:57:38.773271 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a09b827-b7ff-459d-86c9-0ed7abed6d4b admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:38.775321 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3155fab1-900e-4694-9edb-caac45182f31 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:38.778948 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ff3d1fb-01e5-4ece-b112-248e4537f63f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:38.779850 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ff3d1fb-01e5-4ece-b112-248e4537f63f admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:38.780233 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ff3d1fb-01e5-4ece-b112-248e4537f63f admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:38.780613 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ff3d1fb-01e5-4ece-b112-248e4537f63f admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:38.781362 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ff3d1fb-01e5-4ece-b112-248e4537f63f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:38.783331 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ff3d1fb-01e5-4ece-b112-248e4537f63f admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:38.784920 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3155fab1-900e-4694-9edb-caac45182f31 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:38.789631 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-581d8b60-5fdf-4941-8b00-09f88763dd7f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:38.790284 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-581d8b60-5fdf-4941-8b00-09f88763dd7f admin admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:38.790567 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-581d8b60-5fdf-4941-8b00-09f88763dd7f admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:38.790840 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-581d8b60-5fdf-4941-8b00-09f88763dd7f admin admin] PATH_INFO: `/v3/projects/cac43224e7954c44a71839707ffcde6c/users/88f506c18af44d3e8ea2f64bd627d804/roles/092036ee46ac427baf4d56468575f697` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:38.792784 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-581d8b60-5fdf-4941-8b00-09f88763dd7f admin admin] RBAC: Authorizing `identity:create_grant(project_id=cac43224e7954c44a71839707ffcde6c, user_id=88f506c18af44d3e8ea2f64bd627d804, role_id=092036ee46ac427baf4d56468575f697)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:38.794864 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1142/2247] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:57:38 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:57:38.796429 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-581d8b60-5fdf-4941-8b00-09f88763dd7f admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:38.798706 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ff3d1fb-01e5-4ece-b112-248e4537f63f admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:38.810125 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1106/2248] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:57:38 2022] PUT /identity/v3/projects/cac43224e7954c44a71839707ffcde6c/users/88f506c18af44d3e8ea2f64bd627d804/roles/092036ee46ac427baf4d56468575f697 => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 09 00:57:38.811631 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ff3d1fb-01e5-4ece-b112-248e4537f63f admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:38.893228 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22b5bd95-1e2f-4da6-8b46-56011e0b30e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:38.893892 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b5bd95-1e2f-4da6-8b46-56011e0b30e7 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:38.894232 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b5bd95-1e2f-4da6-8b46-56011e0b30e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:38.894511 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b5bd95-1e2f-4da6-8b46-56011e0b30e7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:38.897483 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22b5bd95-1e2f-4da6-8b46-56011e0b30e7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:38.900302 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22b5bd95-1e2f-4da6-8b46-56011e0b30e7 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:38.915649 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1143/2249] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:57:38 2022] GET /identity/v3/roles => generated 1420 bytes in 116 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 09 00:57:38.918467 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-581d8b60-5fdf-4941-8b00-09f88763dd7f admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:38.931314 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-581d8b60-5fdf-4941-8b00-09f88763dd7f admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:38.936614 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e220ed83-2a39-4059-a54e-b3b50d73bb34 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:38.937198 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e220ed83-2a39-4059-a54e-b3b50d73bb34 admin admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:38.938022 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e220ed83-2a39-4059-a54e-b3b50d73bb34 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:38.938022 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e220ed83-2a39-4059-a54e-b3b50d73bb34 admin admin] PATH_INFO: `/v3/projects/6e99f83d1032419d9ecc59913e902c54/users/11c6e683b2b642cd8a11b2506999811f/roles/588016933bab436d8b7c3ac76827356d` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:38.959786 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e220ed83-2a39-4059-a54e-b3b50d73bb34 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6e99f83d1032419d9ecc59913e902c54, user_id=11c6e683b2b642cd8a11b2506999811f, role_id=588016933bab436d8b7c3ac76827356d)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:38.963351 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e220ed83-2a39-4059-a54e-b3b50d73bb34 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:38.981751 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1107/2250] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:57:38 2022] PUT /identity/v3/projects/6e99f83d1032419d9ecc59913e902c54/users/11c6e683b2b642cd8a11b2506999811f/roles/588016933bab436d8b7c3ac76827356d => generated 0 bytes in 64 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:38.992855 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf6196ab-38d2-4241-97d3-d1aa3d64ccdd None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:38.994186 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf6196ab-38d2-4241-97d3-d1aa3d64ccdd None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:38.994563 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf6196ab-38d2-4241-97d3-d1aa3d64ccdd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:39.027098 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-cf6196ab-38d2-4241-97d3-d1aa3d64ccdd None None] MFA Rules not processed for user `11c6e683b2b642cd8a11b2506999811f`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:39.028538 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-cf6196ab-38d2-4241-97d3-d1aa3d64ccdd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:39.090026 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:39.125740 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1144/2251] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:38 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:57:39.137278 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e220ed83-2a39-4059-a54e-b3b50d73bb34 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:39.139163 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-e220ed83-2a39-4059-a54e-b3b50d73bb34 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:39.170112 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:39.235956 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e220ed83-2a39-4059-a54e-b3b50d73bb34 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:39.241120 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab0d6d88-391c-4de7-a4f6-db1e70be0ce4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3f48335668a4e8f934ac11ff66cd308', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:39.241812 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab0d6d88-391c-4de7-a4f6-db1e70be0ce4 service cinder] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:39.242323 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab0d6d88-391c-4de7-a4f6-db1e70be0ce4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:39.242694 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab0d6d88-391c-4de7-a4f6-db1e70be0ce4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:39.255707 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab0d6d88-391c-4de7-a4f6-db1e70be0ce4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:39.258119 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab0d6d88-391c-4de7-a4f6-db1e70be0ce4 service cinder] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:39.271870 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1108/2252] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:57:39 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 135 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:57:44.602590 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b0af537-15e4-40ea-9545-31b108a9d543 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:44.603148 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b0af537-15e4-40ea-9545-31b108a9d543 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:44.603374 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b0af537-15e4-40ea-9545-31b108a9d543 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:44.634054 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-8b0af537-15e4-40ea-9545-31b108a9d543 None None] MFA Rules not processed for user `88f506c18af44d3e8ea2f64bd627d804`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:44.635483 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-8b0af537-15e4-40ea-9545-31b108a9d543 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:44.711866 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1145/2253] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:44 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:57:44.726957 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab0d6d88-391c-4de7-a4f6-db1e70be0ce4 service cinder] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:44.728533 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-ab0d6d88-391c-4de7-a4f6-db1e70be0ce4 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:44.842941 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab0d6d88-391c-4de7-a4f6-db1e70be0ce4 service cinder] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:44.849924 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68919da1-2f75-4619-94b0-7ee1761243f3 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a5d95969e74157bb2d2a57207e1651', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:44.850496 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68919da1-2f75-4619-94b0-7ee1761243f3 service nova] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:44.850817 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68919da1-2f75-4619-94b0-7ee1761243f3 service nova] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:44.851091 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68919da1-2f75-4619-94b0-7ee1761243f3 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:44.852986 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-68919da1-2f75-4619-94b0-7ee1761243f3 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:44.926550 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68919da1-2f75-4619-94b0-7ee1761243f3 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:44.928212 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68919da1-2f75-4619-94b0-7ee1761243f3 service nova] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:44.940055 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1109/2254] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:57:44 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 214 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:57:49.203519 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b0af537-15e4-40ea-9545-31b108a9d543 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:49.205413 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-8b0af537-15e4-40ea-9545-31b108a9d543 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:49.316170 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b0af537-15e4-40ea-9545-31b108a9d543 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:49.323010 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35a47f91-0943-427b-bc54-453dc19ed06a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:49.323761 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a47f91-0943-427b-bc54-453dc19ed06a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:49.324505 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a47f91-0943-427b-bc54-453dc19ed06a admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:49.324505 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a47f91-0943-427b-bc54-453dc19ed06a admin admin] PATH_INFO: `/v3/users/ae420a548f06474db6eb4ca083883a34` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:49.325643 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35a47f91-0943-427b-bc54-453dc19ed06a admin admin] RBAC: Authorizing `identity:delete_user(user_id=ae420a548f06474db6eb4ca083883a34)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:49.327610 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35a47f91-0943-427b-bc54-453dc19ed06a admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:49.400883 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-35a47f91-0943-427b-bc54-453dc19ed06a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ae420a548f06474db6eb4ca083883a34'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:49.410487 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-35a47f91-0943-427b-bc54-453dc19ed06a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ae420a548f06474db6eb4ca083883a34'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:49.411225 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-35a47f91-0943-427b-bc54-453dc19ed06a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ae420a548f06474db6eb4ca083883a34'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:49.411983 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-35a47f91-0943-427b-bc54-453dc19ed06a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ae420a548f06474db6eb4ca083883a34'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:49.421731 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-35a47f91-0943-427b-bc54-453dc19ed06a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ae420a548f06474db6eb4ca083883a34'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:49.431347 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:49.448347 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1146/2255] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 00:57:49 2022] DELETE /identity/v3/users/ae420a548f06474db6eb4ca083883a34 => generated 0 bytes in 245 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:49.453030 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d324aa48-4b44-4913-82ff-d890a4d5f68b None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:49.453654 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d324aa48-4b44-4913-82ff-d890a4d5f68b None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:49.453654 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d324aa48-4b44-4913-82ff-d890a4d5f68b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:49.489370 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-d324aa48-4b44-4913-82ff-d890a4d5f68b None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:49.490625 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-d324aa48-4b44-4913-82ff-d890a4d5f68b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:49.556523 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:49.565358 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1110/2256] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:57:49.576400 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35a47f91-0943-427b-bc54-453dc19ed06a admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:49.577893 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-35a47f91-0943-427b-bc54-453dc19ed06a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:49.635098 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35a47f91-0943-427b-bc54-453dc19ed06a admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:49.638827 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a758b19d-6292-4c57-bd39-c445bc293772 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:49.639434 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a758b19d-6292-4c57-bd39-c445bc293772 service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:49.639748 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a758b19d-6292-4c57-bd39-c445bc293772 service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:49.640026 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a758b19d-6292-4c57-bd39-c445bc293772 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:49.650248 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a758b19d-6292-4c57-bd39-c445bc293772 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:49.651716 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a758b19d-6292-4c57-bd39-c445bc293772 service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:49.661413 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1147/2257] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:57:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:57:49.914505 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d324aa48-4b44-4913-82ff-d890a4d5f68b None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:49.917005 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-d324aa48-4b44-4913-82ff-d890a4d5f68b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:49.935819 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d324aa48-4b44-4913-82ff-d890a4d5f68b None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:49.956493 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4a4bc87-01e3-4713-b368-77f1c6ba5e92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:49.958374 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a4bc87-01e3-4713-b368-77f1c6ba5e92 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:49.958884 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a4bc87-01e3-4713-b368-77f1c6ba5e92 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:49.960227 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a4bc87-01e3-4713-b368-77f1c6ba5e92 admin admin] PATH_INFO: `/v3/users/8acf8140969a4295a04fedd7cde5ab68` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:49.962608 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4a4bc87-01e3-4713-b368-77f1c6ba5e92 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8acf8140969a4295a04fedd7cde5ab68)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:49.965917 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4a4bc87-01e3-4713-b368-77f1c6ba5e92 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:50.036035 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-b4a4bc87-01e3-4713-b368-77f1c6ba5e92 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8acf8140969a4295a04fedd7cde5ab68'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:50.070463 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-b4a4bc87-01e3-4713-b368-77f1c6ba5e92 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8acf8140969a4295a04fedd7cde5ab68'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:50.071389 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-b4a4bc87-01e3-4713-b368-77f1c6ba5e92 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8acf8140969a4295a04fedd7cde5ab68'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:50.088408 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-b4a4bc87-01e3-4713-b368-77f1c6ba5e92 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8acf8140969a4295a04fedd7cde5ab68'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:50.104395 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-b4a4bc87-01e3-4713-b368-77f1c6ba5e92 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8acf8140969a4295a04fedd7cde5ab68'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:50.113393 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1111/2258] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 00:57:49 2022] DELETE /identity/v3/users/8acf8140969a4295a04fedd7cde5ab68 => generated 0 bytes in 199 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 09 00:57:50.119292 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614fd6f4-fc2e-4c49-9d89-88c2a7e36e44 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:50.120048 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614fd6f4-fc2e-4c49-9d89-88c2a7e36e44 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:50.120508 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614fd6f4-fc2e-4c49-9d89-88c2a7e36e44 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:50.180817 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-614fd6f4-fc2e-4c49-9d89-88c2a7e36e44 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:50.182023 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-614fd6f4-fc2e-4c49-9d89-88c2a7e36e44 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:50.317522 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1148/2259] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 199 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:57:50.329361 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4a4bc87-01e3-4713-b368-77f1c6ba5e92 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:50.331556 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-b4a4bc87-01e3-4713-b368-77f1c6ba5e92 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:50.417362 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4a4bc87-01e3-4713-b368-77f1c6ba5e92 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:50.422617 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7b95974-552d-45b3-9c99-5fb970972ab8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:50.423308 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b95974-552d-45b3-9c99-5fb970972ab8 service neutron] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:50.423626 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b95974-552d-45b3-9c99-5fb970972ab8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:50.423885 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b95974-552d-45b3-9c99-5fb970972ab8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:50.436762 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7b95974-552d-45b3-9c99-5fb970972ab8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:50.440650 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7b95974-552d-45b3-9c99-5fb970972ab8 service neutron] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:50.459816 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-614fd6f4-fc2e-4c49-9d89-88c2a7e36e44 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:50.460664 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1112/2260] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:57:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 133 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 09 00:57:50.461798 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-614fd6f4-fc2e-4c49-9d89-88c2a7e36e44 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:50.475753 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-614fd6f4-fc2e-4c49-9d89-88c2a7e36e44 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:50.481293 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98601ed6-e8ff-4b3e-96e1-41ad2a92abd8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:50.482148 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98601ed6-e8ff-4b3e-96e1-41ad2a92abd8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:50.483295 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98601ed6-e8ff-4b3e-96e1-41ad2a92abd8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:50.483809 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98601ed6-e8ff-4b3e-96e1-41ad2a92abd8 admin admin] PATH_INFO: `/v3/projects/3ffe1ca49c4140ea8b6eecc9d2667b3b` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:50.485775 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98601ed6-e8ff-4b3e-96e1-41ad2a92abd8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3ffe1ca49c4140ea8b6eecc9d2667b3b)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:50.488677 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98601ed6-e8ff-4b3e-96e1-41ad2a92abd8 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:50.555496 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-98601ed6-e8ff-4b3e-96e1-41ad2a92abd8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3ffe1ca49c4140ea8b6eecc9d2667b3b', 'request_id': 'req-98601ed6-e8ff-4b3e-96e1-41ad2a92abd8'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:50.562102 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-98601ed6-e8ff-4b3e-96e1-41ad2a92abd8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3ffe1ca49c4140ea8b6eecc9d2667b3b', 'request_id': 'req-98601ed6-e8ff-4b3e-96e1-41ad2a92abd8'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:50.570507 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-98601ed6-e8ff-4b3e-96e1-41ad2a92abd8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3ffe1ca49c4140ea8b6eecc9d2667b3b', 'request_id': 'req-98601ed6-e8ff-4b3e-96e1-41ad2a92abd8'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:50.586784 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-98601ed6-e8ff-4b3e-96e1-41ad2a92abd8 admin admin] The token cache is being invalidate because project 3ffe1ca49c4140ea8b6eecc9d2667b3b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 00:57:50.587074 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-98601ed6-e8ff-4b3e-96e1-41ad2a92abd8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:50.587795 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-98601ed6-e8ff-4b3e-96e1-41ad2a92abd8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:50.589802 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1149/2261] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 00:57:50 2022] DELETE /identity/v3/projects/3ffe1ca49c4140ea8b6eecc9d2667b3b => generated 0 bytes in 131 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:50.595807 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efa9ee97-de6a-45af-b4b6-b7fe422a8302 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:50.596227 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efa9ee97-de6a-45af-b4b6-b7fe422a8302 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:50.596629 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efa9ee97-de6a-45af-b4b6-b7fe422a8302 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:50.626666 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-efa9ee97-de6a-45af-b4b6-b7fe422a8302 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:50.628152 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-efa9ee97-de6a-45af-b4b6-b7fe422a8302 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:50.728684 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1113/2262] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:57:50.736576 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5502a348-4c45-45fc-9f35-ca91026747df None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:50.738443 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5502a348-4c45-45fc-9f35-ca91026747df None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:50.738931 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5502a348-4c45-45fc-9f35-ca91026747df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:50.765365 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-5502a348-4c45-45fc-9f35-ca91026747df None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:50.766607 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-5502a348-4c45-45fc-9f35-ca91026747df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:50.786624 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1150/2263] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 51 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:57:50.787859 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efa9ee97-de6a-45af-b4b6-b7fe422a8302 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:50.800708 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efa9ee97-de6a-45af-b4b6-b7fe422a8302 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:50.808173 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89cba255-fa6d-4c68-bdd2-72a59e56be7a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:50.809126 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89cba255-fa6d-4c68-bdd2-72a59e56be7a admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:50.809630 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89cba255-fa6d-4c68-bdd2-72a59e56be7a admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:50.810512 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89cba255-fa6d-4c68-bdd2-72a59e56be7a admin admin] PATH_INFO: `/v3/domains` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:50.811365 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89cba255-fa6d-4c68-bdd2-72a59e56be7a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:57:50.811910 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89cba255-fa6d-4c68-bdd2-72a59e56be7a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:50.814009 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89cba255-fa6d-4c68-bdd2-72a59e56be7a admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:50.833820 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1114/2264] 10.222.0.33 () {58 vars in 1153 bytes} [Sun Oct 9 00:57:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:57:50.839386 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5502a348-4c45-45fc-9f35-ca91026747df None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:50.858897 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5502a348-4c45-45fc-9f35-ca91026747df None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:50.864841 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-596dc0d7-d257-462a-8c11-1316af3aa4fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:50.865713 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-596dc0d7-d257-462a-8c11-1316af3aa4fb admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:50.866193 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-596dc0d7-d257-462a-8c11-1316af3aa4fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:50.866698 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-596dc0d7-d257-462a-8c11-1316af3aa4fb admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:50.867535 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-596dc0d7-d257-462a-8c11-1316af3aa4fb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:50.870414 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-596dc0d7-d257-462a-8c11-1316af3aa4fb admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:50.876649 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:50.913376 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1151/2265] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 00:57:50 2022] POST /identity/v3/projects => generated 371 bytes in 75 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:57:50.913902 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89cba255-fa6d-4c68-bdd2-72a59e56be7a admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:50.920130 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:50.926678 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89cba255-fa6d-4c68-bdd2-72a59e56be7a admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:50.996464 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d05589e-eafd-4fa6-8a07-967cdadfe240 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:50.997170 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d05589e-eafd-4fa6-8a07-967cdadfe240 admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:50.997538 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d05589e-eafd-4fa6-8a07-967cdadfe240 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:50.997876 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d05589e-eafd-4fa6-8a07-967cdadfe240 admin admin] PATH_INFO: `/v3/users` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:50.998582 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d05589e-eafd-4fa6-8a07-967cdadfe240 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:51.000563 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d05589e-eafd-4fa6-8a07-967cdadfe240 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:51.022929 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1115/2266] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:57:50 2022] POST /identity/v3/users => generated 286 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 09 00:57:51.027171 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-596dc0d7-d257-462a-8c11-1316af3aa4fb admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:51.039332 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-596dc0d7-d257-462a-8c11-1316af3aa4fb admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:51.044731 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c2a63e9-ca3d-45b4-8ca7-d33b6ede4f26 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:51.045127 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c2a63e9-ca3d-45b4-8ca7-d33b6ede4f26 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:51.045541 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c2a63e9-ca3d-45b4-8ca7-d33b6ede4f26 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:51.046037 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c2a63e9-ca3d-45b4-8ca7-d33b6ede4f26 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:51.046781 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c2a63e9-ca3d-45b4-8ca7-d33b6ede4f26 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:51.048520 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c2a63e9-ca3d-45b4-8ca7-d33b6ede4f26 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:51.059392 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1152/2267] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:57:51 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:57:51.063116 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d05589e-eafd-4fa6-8a07-967cdadfe240 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:51.074485 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d05589e-eafd-4fa6-8a07-967cdadfe240 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:51.079611 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b744902f-bc19-4b27-9905-4c90200f7421 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:51.080229 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b744902f-bc19-4b27-9905-4c90200f7421 admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:51.080519 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b744902f-bc19-4b27-9905-4c90200f7421 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:51.080792 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b744902f-bc19-4b27-9905-4c90200f7421 admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:51.081513 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b744902f-bc19-4b27-9905-4c90200f7421 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:51.083749 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b744902f-bc19-4b27-9905-4c90200f7421 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:51.094346 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c2a63e9-ca3d-45b4-8ca7-d33b6ede4f26 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:51.096620 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-6c2a63e9-ca3d-45b4-8ca7-d33b6ede4f26 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:51.102012 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1116/2268] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:57:51 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 09 00:57:51.103436 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b744902f-bc19-4b27-9905-4c90200f7421 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:51.114683 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b744902f-bc19-4b27-9905-4c90200f7421 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:51.114898 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c2a63e9-ca3d-45b4-8ca7-d33b6ede4f26 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:51.120733 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16ce5ff1-457f-4804-a17a-9603c90fcfab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:51.120919 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ff77417-3787-4803-a70e-089453bfb8cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:51.121515 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ff77417-3787-4803-a70e-089453bfb8cb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:51.121627 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16ce5ff1-457f-4804-a17a-9603c90fcfab admin admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:51.121796 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16ce5ff1-457f-4804-a17a-9603c90fcfab admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:51.121910 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ff77417-3787-4803-a70e-089453bfb8cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:51.122076 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16ce5ff1-457f-4804-a17a-9603c90fcfab admin admin] PATH_INFO: `/v3/projects/855bbf7d882d4e88a30ff3b1d83dd4af/users/4a73ba10f87a49dfa3f9abca93819dea/roles/588016933bab436d8b7c3ac76827356d` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:51.122862 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ff77417-3787-4803-a70e-089453bfb8cb admin admin] PATH_INFO: `/v3/projects/95f6d79f28744858b8cfe8c8313c2ff2` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:51.124124 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ff77417-3787-4803-a70e-089453bfb8cb admin admin] RBAC: Authorizing `identity:delete_project(project_id=95f6d79f28744858b8cfe8c8313c2ff2)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:51.126693 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ff77417-3787-4803-a70e-089453bfb8cb admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:51.142035 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16ce5ff1-457f-4804-a17a-9603c90fcfab admin admin] RBAC: Authorizing `identity:create_grant(project_id=855bbf7d882d4e88a30ff3b1d83dd4af, user_id=4a73ba10f87a49dfa3f9abca93819dea, role_id=588016933bab436d8b7c3ac76827356d)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:51.145380 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16ce5ff1-457f-4804-a17a-9603c90fcfab admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:51.167964 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1117/2269] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:57:51 2022] PUT /identity/v3/projects/855bbf7d882d4e88a30ff3b1d83dd4af/users/4a73ba10f87a49dfa3f9abca93819dea/roles/588016933bab436d8b7c3ac76827356d => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 09 00:57:51.175548 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a45cfe-cda7-4341-b557-c59bdad9ba5c None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:51.175940 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a45cfe-cda7-4341-b557-c59bdad9ba5c None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:51.176310 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a45cfe-cda7-4341-b557-c59bdad9ba5c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:51.213256 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-2ff77417-3787-4803-a70e-089453bfb8cb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '95f6d79f28744858b8cfe8c8313c2ff2', 'request_id': 'req-2ff77417-3787-4803-a70e-089453bfb8cb'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:51.215453 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-46a45cfe-cda7-4341-b557-c59bdad9ba5c None None] MFA Rules not processed for user `4a73ba10f87a49dfa3f9abca93819dea`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:51.216781 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-46a45cfe-cda7-4341-b557-c59bdad9ba5c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:51.221524 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-2ff77417-3787-4803-a70e-089453bfb8cb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '95f6d79f28744858b8cfe8c8313c2ff2', 'request_id': 'req-2ff77417-3787-4803-a70e-089453bfb8cb'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:51.232160 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-2ff77417-3787-4803-a70e-089453bfb8cb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '95f6d79f28744858b8cfe8c8313c2ff2', 'request_id': 'req-2ff77417-3787-4803-a70e-089453bfb8cb'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:51.246836 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-2ff77417-3787-4803-a70e-089453bfb8cb admin admin] The token cache is being invalidate because project 95f6d79f28744858b8cfe8c8313c2ff2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 00:57:51.247548 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-2ff77417-3787-4803-a70e-089453bfb8cb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:51.248770 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-2ff77417-3787-4803-a70e-089453bfb8cb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:51.251273 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1153/2270] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 00:57:51 2022] DELETE /identity/v3/projects/95f6d79f28744858b8cfe8c8313c2ff2 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 09 00:57:51.256913 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ff77417-3787-4803-a70e-089453bfb8cb admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:51.258748 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-2ff77417-3787-4803-a70e-089453bfb8cb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:51.296241 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1118/2271] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:51 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 09 00:57:51.311291 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46a45cfe-cda7-4341-b557-c59bdad9ba5c None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:51.311291 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-46a45cfe-cda7-4341-b557-c59bdad9ba5c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:51.342332 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ff77417-3787-4803-a70e-089453bfb8cb admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:51.347839 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd2010ef-cbf9-4ef3-820c-01c44040475c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:51.348956 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd2010ef-cbf9-4ef3-820c-01c44040475c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:51.352103 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd2010ef-cbf9-4ef3-820c-01c44040475c admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:51.352103 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd2010ef-cbf9-4ef3-820c-01c44040475c admin admin] PATH_INFO: `/v3/users/e8f4849ef4bd428b9489acabdda54b95` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:51.353069 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd2010ef-cbf9-4ef3-820c-01c44040475c admin admin] RBAC: Authorizing `identity:delete_user(user_id=e8f4849ef4bd428b9489acabdda54b95)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:51.355003 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd2010ef-cbf9-4ef3-820c-01c44040475c admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:51.380187 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:51.416578 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46a45cfe-cda7-4341-b557-c59bdad9ba5c None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:51.422444 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-dd2010ef-cbf9-4ef3-820c-01c44040475c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e8f4849ef4bd428b9489acabdda54b95'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:51.422551 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1aafce4-856d-45a4-bdaa-f9d11677b6c2 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1f7a88094d73471787cc0a287adbd394', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:51.422971 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1aafce4-856d-45a4-bdaa-f9d11677b6c2 service glance] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:51.423257 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1aafce4-856d-45a4-bdaa-f9d11677b6c2 service glance] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:51.423535 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1aafce4-856d-45a4-bdaa-f9d11677b6c2 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:51.431828 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-dd2010ef-cbf9-4ef3-820c-01c44040475c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e8f4849ef4bd428b9489acabdda54b95'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:51.432696 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-dd2010ef-cbf9-4ef3-820c-01c44040475c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e8f4849ef4bd428b9489acabdda54b95'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:51.433335 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-dd2010ef-cbf9-4ef3-820c-01c44040475c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e8f4849ef4bd428b9489acabdda54b95'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:51.433829 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1aafce4-856d-45a4-bdaa-f9d11677b6c2 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:51.435496 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1aafce4-856d-45a4-bdaa-f9d11677b6c2 service glance] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:51.437780 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=82,keystone:INSERT=3,keystone:DELETE=15 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:51.444180 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-dd2010ef-cbf9-4ef3-820c-01c44040475c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e8f4849ef4bd428b9489acabdda54b95'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:51.450016 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1119/2272] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:57:51 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 141 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:57:51.474271 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1154/2273] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 00:57:51 2022] DELETE /identity/v3/users/e8f4849ef4bd428b9489acabdda54b95 => generated 0 bytes in 219 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 09 00:57:51.871513 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1aafce4-856d-45a4-bdaa-f9d11677b6c2 service glance] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:51.883388 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1aafce4-856d-45a4-bdaa-f9d11677b6c2 service glance] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:51.953191 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92f5e92f-9bec-4c8f-9423-32abf5a518cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:51.953828 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92f5e92f-9bec-4c8f-9423-32abf5a518cd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:51.954111 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92f5e92f-9bec-4c8f-9423-32abf5a518cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:51.954390 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92f5e92f-9bec-4c8f-9423-32abf5a518cd admin admin] PATH_INFO: `/v3/projects/a242229ba2ba4d10b765848a9cdd89a5` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:51.955531 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92f5e92f-9bec-4c8f-9423-32abf5a518cd admin admin] RBAC: Authorizing `identity:delete_project(project_id=a242229ba2ba4d10b765848a9cdd89a5)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:51.958282 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92f5e92f-9bec-4c8f-9423-32abf5a518cd admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:52.024908 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-92f5e92f-9bec-4c8f-9423-32abf5a518cd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a242229ba2ba4d10b765848a9cdd89a5', 'request_id': 'req-92f5e92f-9bec-4c8f-9423-32abf5a518cd'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:52.030328 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-92f5e92f-9bec-4c8f-9423-32abf5a518cd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a242229ba2ba4d10b765848a9cdd89a5', 'request_id': 'req-92f5e92f-9bec-4c8f-9423-32abf5a518cd'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:52.038296 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-92f5e92f-9bec-4c8f-9423-32abf5a518cd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a242229ba2ba4d10b765848a9cdd89a5', 'request_id': 'req-92f5e92f-9bec-4c8f-9423-32abf5a518cd'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:52.053325 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-92f5e92f-9bec-4c8f-9423-32abf5a518cd admin admin] The token cache is being invalidate because project a242229ba2ba4d10b765848a9cdd89a5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81453) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 00:57:52.053615 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-92f5e92f-9bec-4c8f-9423-32abf5a518cd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:52.054273 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-92f5e92f-9bec-4c8f-9423-32abf5a518cd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:52.055879 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1120/2274] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 00:57:51 2022] DELETE /identity/v3/projects/a242229ba2ba4d10b765848a9cdd89a5 => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:52.060718 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e5d76a7-e92b-4023-8b97-1924c84d4012 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:52.061081 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e5d76a7-e92b-4023-8b97-1924c84d4012 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:52.061395 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e5d76a7-e92b-4023-8b97-1924c84d4012 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:52.085058 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-1e5d76a7-e92b-4023-8b97-1924c84d4012 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:52.086638 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-1e5d76a7-e92b-4023-8b97-1924c84d4012 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:52.145025 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1155/2275] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:57:52.152840 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974fe9dc-4bcd-48b7-a4a8-bf0c790a7a71 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:52.153128 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974fe9dc-4bcd-48b7-a4a8-bf0c790a7a71 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:52.153407 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974fe9dc-4bcd-48b7-a4a8-bf0c790a7a71 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:52.174316 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-974fe9dc-4bcd-48b7-a4a8-bf0c790a7a71 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:52.176065 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-974fe9dc-4bcd-48b7-a4a8-bf0c790a7a71 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:52.184196 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1121/2276] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:57:52.188288 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e5d76a7-e92b-4023-8b97-1924c84d4012 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:52.198201 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e5d76a7-e92b-4023-8b97-1924c84d4012 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:52.202671 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8399ba70-1e4c-4133-b1c0-fb29b82cf80d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:52.203253 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8399ba70-1e4c-4133-b1c0-fb29b82cf80d admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:52.203525 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8399ba70-1e4c-4133-b1c0-fb29b82cf80d admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:52.203825 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8399ba70-1e4c-4133-b1c0-fb29b82cf80d admin admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:52.204455 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8399ba70-1e4c-4133-b1c0-fb29b82cf80d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:57:52.204751 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8399ba70-1e4c-4133-b1c0-fb29b82cf80d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:52.206362 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8399ba70-1e4c-4133-b1c0-fb29b82cf80d admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:52.217961 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1156/2277] 10.222.0.33 () {58 vars in 1153 bytes} [Sun Oct 9 00:57:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:57:52.222190 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-974fe9dc-4bcd-48b7-a4a8-bf0c790a7a71 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:52.231275 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-974fe9dc-4bcd-48b7-a4a8-bf0c790a7a71 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:52.235854 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aad50eb1-aed7-4825-9ffd-2a617b1e06e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:52.236436 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aad50eb1-aed7-4825-9ffd-2a617b1e06e7 admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:52.236710 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aad50eb1-aed7-4825-9ffd-2a617b1e06e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:52.236976 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aad50eb1-aed7-4825-9ffd-2a617b1e06e7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:52.237604 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aad50eb1-aed7-4825-9ffd-2a617b1e06e7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:52.240044 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aad50eb1-aed7-4825-9ffd-2a617b1e06e7 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:52.266726 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1122/2278] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 00:57:52 2022] POST /identity/v3/projects => generated 399 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 09 00:57:52.270876 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8399ba70-1e4c-4133-b1c0-fb29b82cf80d admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:52.281105 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8399ba70-1e4c-4133-b1c0-fb29b82cf80d admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:52.337507 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8081443a-e2ff-4549-8184-319e6501194a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:52.338104 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8081443a-e2ff-4549-8184-319e6501194a admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:52.338385 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8081443a-e2ff-4549-8184-319e6501194a admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:52.338656 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8081443a-e2ff-4549-8184-319e6501194a admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:52.339274 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8081443a-e2ff-4549-8184-319e6501194a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:52.341218 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8081443a-e2ff-4549-8184-319e6501194a admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:52.361639 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1157/2279] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:57:52 2022] POST /identity/v3/users => generated 300 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:57:52.366023 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aad50eb1-aed7-4825-9ffd-2a617b1e06e7 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:52.376359 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aad50eb1-aed7-4825-9ffd-2a617b1e06e7 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:52.381545 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2b40a8e-8a33-448a-b102-75ab26a537b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:52.382138 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2b40a8e-8a33-448a-b102-75ab26a537b0 admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:52.382417 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2b40a8e-8a33-448a-b102-75ab26a537b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:52.382715 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2b40a8e-8a33-448a-b102-75ab26a537b0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:52.383371 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2b40a8e-8a33-448a-b102-75ab26a537b0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:52.384974 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2b40a8e-8a33-448a-b102-75ab26a537b0 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:52.394637 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1123/2280] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:57:52 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:57:52.398429 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8081443a-e2ff-4549-8184-319e6501194a admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:52.408622 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8081443a-e2ff-4549-8184-319e6501194a admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:52.413407 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b73e15a7-452f-44c5-a23a-fb25f38fbf72 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:52.414003 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b73e15a7-452f-44c5-a23a-fb25f38fbf72 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:52.414286 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b73e15a7-452f-44c5-a23a-fb25f38fbf72 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:52.414621 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b73e15a7-452f-44c5-a23a-fb25f38fbf72 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:52.415324 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b73e15a7-452f-44c5-a23a-fb25f38fbf72 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:52.417218 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b73e15a7-452f-44c5-a23a-fb25f38fbf72 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:52.431367 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1158/2281] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:57:52 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:57:52.435400 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2b40a8e-8a33-448a-b102-75ab26a537b0 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:52.445809 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2b40a8e-8a33-448a-b102-75ab26a537b0 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:52.450939 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d498daef-f031-48f6-9e2e-f788dcc63444 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:52.451658 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d498daef-f031-48f6-9e2e-f788dcc63444 admin admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:52.451910 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d498daef-f031-48f6-9e2e-f788dcc63444 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:52.452183 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d498daef-f031-48f6-9e2e-f788dcc63444 admin admin] PATH_INFO: `/v3/projects/12ae8ab3a3a64ff6887430aab9669232/users/92853af5786743dc82d0223c088b401c/roles/588016933bab436d8b7c3ac76827356d` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:52.472524 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d498daef-f031-48f6-9e2e-f788dcc63444 admin admin] RBAC: Authorizing `identity:create_grant(project_id=12ae8ab3a3a64ff6887430aab9669232, user_id=92853af5786743dc82d0223c088b401c, role_id=588016933bab436d8b7c3ac76827356d)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:52.475279 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d498daef-f031-48f6-9e2e-f788dcc63444 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:52.478671 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:52.493477 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1124/2282] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:57:52 2022] PUT /identity/v3/projects/12ae8ab3a3a64ff6887430aab9669232/users/92853af5786743dc82d0223c088b401c/roles/588016933bab436d8b7c3ac76827356d => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 09 00:57:52.505570 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa6f5eff-bfaa-4392-a1f2-42d5ce8acea3 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:52.505855 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa6f5eff-bfaa-4392-a1f2-42d5ce8acea3 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:52.506129 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa6f5eff-bfaa-4392-a1f2-42d5ce8acea3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:52.533553 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-fa6f5eff-bfaa-4392-a1f2-42d5ce8acea3 None None] MFA Rules not processed for user `92853af5786743dc82d0223c088b401c`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:52.535270 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-fa6f5eff-bfaa-4392-a1f2-42d5ce8acea3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:52.567527 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=95,keystone:DELETE=2,keystone:INSERT=3 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:52.601470 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1159/2283] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:52 2022] POST /identity/v3/auth/tokens => generated 3276 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:57:52.605919 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d498daef-f031-48f6-9e2e-f788dcc63444 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:52.667323 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d498daef-f031-48f6-9e2e-f788dcc63444 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:52.672705 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e3aa25-2178-4128-b25e-b6701d6b1b33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:52.673309 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e3aa25-2178-4128-b25e-b6701d6b1b33 admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:52.673591 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e3aa25-2178-4128-b25e-b6701d6b1b33 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:52.673863 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e3aa25-2178-4128-b25e-b6701d6b1b33 admin admin] PATH_INFO: `/v3/projects` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:52.674502 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0e3aa25-2178-4128-b25e-b6701d6b1b33 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:52.677017 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0e3aa25-2178-4128-b25e-b6701d6b1b33 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:52.703698 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1125/2284] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 00:57:52 2022] POST /identity/v3/projects => generated 401 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 09 00:57:52.707689 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa6f5eff-bfaa-4392-a1f2-42d5ce8acea3 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:52.718237 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa6f5eff-bfaa-4392-a1f2-42d5ce8acea3 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:52.776309 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65589927-322e-4296-bf57-7529038ecefd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:52.777021 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65589927-322e-4296-bf57-7529038ecefd admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:52.777308 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65589927-322e-4296-bf57-7529038ecefd admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:52.777581 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65589927-322e-4296-bf57-7529038ecefd admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:52.778213 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65589927-322e-4296-bf57-7529038ecefd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:52.780228 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65589927-322e-4296-bf57-7529038ecefd admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:52.807639 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1160/2285] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:57:52 2022] POST /identity/v3/users => generated 307 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:57:52.812738 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e3aa25-2178-4128-b25e-b6701d6b1b33 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:52.823472 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e3aa25-2178-4128-b25e-b6701d6b1b33 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:52.828200 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f50fd4ad-074d-49e7-a16c-96661aad43eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:52.829014 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f50fd4ad-074d-49e7-a16c-96661aad43eb admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:52.829406 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f50fd4ad-074d-49e7-a16c-96661aad43eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:52.829779 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f50fd4ad-074d-49e7-a16c-96661aad43eb admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:52.830613 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f50fd4ad-074d-49e7-a16c-96661aad43eb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:52.832297 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f50fd4ad-074d-49e7-a16c-96661aad43eb admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:52.843251 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1126/2286] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:57:52 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:57:52.848197 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65589927-322e-4296-bf57-7529038ecefd admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:52.859637 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65589927-322e-4296-bf57-7529038ecefd admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:52.864786 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f698a0a-f1da-4a46-a08d-be22a6705cfd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:52.865462 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f698a0a-f1da-4a46-a08d-be22a6705cfd admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:52.865744 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f698a0a-f1da-4a46-a08d-be22a6705cfd admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:52.866022 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f698a0a-f1da-4a46-a08d-be22a6705cfd admin admin] PATH_INFO: `/v3/projects/211862bbf0dd4e7092c1c33f6321459c/users/a1cf02e41cf94286bf6e0d48ca343d47/roles/092036ee46ac427baf4d56468575f697` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:52.880576 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f698a0a-f1da-4a46-a08d-be22a6705cfd admin admin] RBAC: Authorizing `identity:create_grant(project_id=211862bbf0dd4e7092c1c33f6321459c, user_id=a1cf02e41cf94286bf6e0d48ca343d47, role_id=092036ee46ac427baf4d56468575f697)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:52.883234 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f698a0a-f1da-4a46-a08d-be22a6705cfd admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:52.896035 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1161/2287] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:57:52 2022] PUT /identity/v3/projects/211862bbf0dd4e7092c1c33f6321459c/users/a1cf02e41cf94286bf6e0d48ca343d47/roles/092036ee46ac427baf4d56468575f697 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:52.900061 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f50fd4ad-074d-49e7-a16c-96661aad43eb admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:52.910838 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f50fd4ad-074d-49e7-a16c-96661aad43eb admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:52.967226 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56e03f07-18a2-43db-961f-487ef4edc954 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:52.967792 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e03f07-18a2-43db-961f-487ef4edc954 admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:52.968072 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e03f07-18a2-43db-961f-487ef4edc954 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:52.968344 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e03f07-18a2-43db-961f-487ef4edc954 admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:52.968973 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56e03f07-18a2-43db-961f-487ef4edc954 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:52.970545 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56e03f07-18a2-43db-961f-487ef4edc954 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:52.980608 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1127/2288] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:57:52 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 09 00:57:52.984792 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f698a0a-f1da-4a46-a08d-be22a6705cfd admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:52.995036 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f698a0a-f1da-4a46-a08d-be22a6705cfd admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:52.999744 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7234449-e674-45b7-b45d-8ad2658c12a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:53.000418 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7234449-e674-45b7-b45d-8ad2658c12a2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:53.000697 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7234449-e674-45b7-b45d-8ad2658c12a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:53.000971 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7234449-e674-45b7-b45d-8ad2658c12a2 admin admin] PATH_INFO: `/v3/projects/211862bbf0dd4e7092c1c33f6321459c/users/a1cf02e41cf94286bf6e0d48ca343d47/roles/092036ee46ac427baf4d56468575f697` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:53.002873 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7234449-e674-45b7-b45d-8ad2658c12a2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=211862bbf0dd4e7092c1c33f6321459c, user_id=a1cf02e41cf94286bf6e0d48ca343d47, role_id=092036ee46ac427baf4d56468575f697)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:53.005545 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7234449-e674-45b7-b45d-8ad2658c12a2 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:53.016279 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1162/2289] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:57:52 2022] PUT /identity/v3/projects/211862bbf0dd4e7092c1c33f6321459c/users/a1cf02e41cf94286bf6e0d48ca343d47/roles/092036ee46ac427baf4d56468575f697 => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:53.026571 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-238919b6-656f-40ce-8250-ad0492dc0281 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:53.026862 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-238919b6-656f-40ce-8250-ad0492dc0281 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:53.027142 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-238919b6-656f-40ce-8250-ad0492dc0281 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:53.055967 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-238919b6-656f-40ce-8250-ad0492dc0281 None None] MFA Rules not processed for user `a1cf02e41cf94286bf6e0d48ca343d47`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:53.057176 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-238919b6-656f-40ce-8250-ad0492dc0281 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:53.109469 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:53.127746 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1128/2290] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:53 2022] POST /identity/v3/auth/tokens => generated 3345 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:57:53.138709 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7234449-e674-45b7-b45d-8ad2658c12a2 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:53.140507 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-d7234449-e674-45b7-b45d-8ad2658c12a2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:53.197944 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7234449-e674-45b7-b45d-8ad2658c12a2 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:53.201515 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6a039e4-ced1-4981-822c-020861bbf90d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3f48335668a4e8f934ac11ff66cd308', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:53.202106 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a039e4-ced1-4981-822c-020861bbf90d service cinder] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:53.202380 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a039e4-ced1-4981-822c-020861bbf90d service cinder] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:53.202649 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a039e4-ced1-4981-822c-020861bbf90d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:53.211913 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6a039e4-ced1-4981-822c-020861bbf90d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:53.213331 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6a039e4-ced1-4981-822c-020861bbf90d service cinder] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:53.223070 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1163/2291] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:57:53 2022] GET /identity/v3/auth/tokens => generated 3345 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:57:53.299704 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-238919b6-656f-40ce-8250-ad0492dc0281 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:53.301106 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-238919b6-656f-40ce-8250-ad0492dc0281 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:53.313691 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-238919b6-656f-40ce-8250-ad0492dc0281 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:53.317806 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aee0aa25-f467-41f7-8ee1-561c78558b6a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3f48335668a4e8f934ac11ff66cd308', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:53.318397 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee0aa25-f467-41f7-8ee1-561c78558b6a service cinder] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:53.318675 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee0aa25-f467-41f7-8ee1-561c78558b6a service cinder] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:53.318941 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aee0aa25-f467-41f7-8ee1-561c78558b6a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:53.377204 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aee0aa25-f467-41f7-8ee1-561c78558b6a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:53.379220 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aee0aa25-f467-41f7-8ee1-561c78558b6a service cinder] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:53.388375 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1129/2292] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:57:53 2022] GET /identity/v3/auth/tokens => generated 3276 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:57:56.845565 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6a039e4-ced1-4981-822c-020861bbf90d service cinder] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:56.847323 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-b6a039e4-ced1-4981-822c-020861bbf90d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:56.882105 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:56.916353 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6a039e4-ced1-4981-822c-020861bbf90d service cinder] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:56.920803 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a291834-ad6e-44a3-bbf9-6ee6a97bbda2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:56.921416 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a291834-ad6e-44a3-bbf9-6ee6a97bbda2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:56.921694 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a291834-ad6e-44a3-bbf9-6ee6a97bbda2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:56.921996 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a291834-ad6e-44a3-bbf9-6ee6a97bbda2 admin admin] PATH_INFO: `/v3/users/11c6e683b2b642cd8a11b2506999811f` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:56.923609 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a291834-ad6e-44a3-bbf9-6ee6a97bbda2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=11c6e683b2b642cd8a11b2506999811f)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:56.926212 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a291834-ad6e-44a3-bbf9-6ee6a97bbda2 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:56.970816 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-1a291834-ad6e-44a3-bbf9-6ee6a97bbda2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '11c6e683b2b642cd8a11b2506999811f'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:56.977753 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-1a291834-ad6e-44a3-bbf9-6ee6a97bbda2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '11c6e683b2b642cd8a11b2506999811f'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:56.978428 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-1a291834-ad6e-44a3-bbf9-6ee6a97bbda2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '11c6e683b2b642cd8a11b2506999811f'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:56.979035 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-1a291834-ad6e-44a3-bbf9-6ee6a97bbda2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '11c6e683b2b642cd8a11b2506999811f'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:56.986864 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-1a291834-ad6e-44a3-bbf9-6ee6a97bbda2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '11c6e683b2b642cd8a11b2506999811f'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:57.005495 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1164/2293] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 00:57:56 2022] DELETE /identity/v3/users/11c6e683b2b642cd8a11b2506999811f => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:57.010076 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e614d864-8022-4799-9495-e72fd77c9f16 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:57.010564 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e614d864-8022-4799-9495-e72fd77c9f16 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:57.010656 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e614d864-8022-4799-9495-e72fd77c9f16 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:57.029308 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-e614d864-8022-4799-9495-e72fd77c9f16 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:57.030512 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-e614d864-8022-4799-9495-e72fd77c9f16 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:57.104083 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1130/2294] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:57:57.114258 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a291834-ad6e-44a3-bbf9-6ee6a97bbda2 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:57.115699 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-1a291834-ad6e-44a3-bbf9-6ee6a97bbda2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:57.173010 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a291834-ad6e-44a3-bbf9-6ee6a97bbda2 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:57.177233 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b028d0ef-d7c4-46a3-b801-a4a16e18555c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:57.177899 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b028d0ef-d7c4-46a3-b801-a4a16e18555c service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:57.178204 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b028d0ef-d7c4-46a3-b801-a4a16e18555c service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:57.178476 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b028d0ef-d7c4-46a3-b801-a4a16e18555c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:57.187549 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b028d0ef-d7c4-46a3-b801-a4a16e18555c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:57.189001 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b028d0ef-d7c4-46a3-b801-a4a16e18555c service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:57.199823 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1165/2295] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:57:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:57:57.596831 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e614d864-8022-4799-9495-e72fd77c9f16 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:57.598803 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-e614d864-8022-4799-9495-e72fd77c9f16 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:57.610407 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e614d864-8022-4799-9495-e72fd77c9f16 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:57.614932 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-261b952b-8a86-443e-86c4-3587807d7288 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:57.615549 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-261b952b-8a86-443e-86c4-3587807d7288 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:57.615867 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-261b952b-8a86-443e-86c4-3587807d7288 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:57.616140 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-261b952b-8a86-443e-86c4-3587807d7288 admin admin] PATH_INFO: `/v3/projects/6e99f83d1032419d9ecc59913e902c54` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:57.617325 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-261b952b-8a86-443e-86c4-3587807d7288 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6e99f83d1032419d9ecc59913e902c54)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:57.619197 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-261b952b-8a86-443e-86c4-3587807d7288 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:57.683099 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-261b952b-8a86-443e-86c4-3587807d7288 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6e99f83d1032419d9ecc59913e902c54', 'request_id': 'req-261b952b-8a86-443e-86c4-3587807d7288'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:57.690935 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-261b952b-8a86-443e-86c4-3587807d7288 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6e99f83d1032419d9ecc59913e902c54', 'request_id': 'req-261b952b-8a86-443e-86c4-3587807d7288'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:57.697723 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-261b952b-8a86-443e-86c4-3587807d7288 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6e99f83d1032419d9ecc59913e902c54', 'request_id': 'req-261b952b-8a86-443e-86c4-3587807d7288'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:57.714092 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-261b952b-8a86-443e-86c4-3587807d7288 admin admin] The token cache is being invalidate because project 6e99f83d1032419d9ecc59913e902c54 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81453) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 00:57:57.714766 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-261b952b-8a86-443e-86c4-3587807d7288 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:57.715525 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-261b952b-8a86-443e-86c4-3587807d7288 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:57.717256 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1131/2296] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 00:57:57 2022] DELETE /identity/v3/projects/6e99f83d1032419d9ecc59913e902c54 => generated 0 bytes in 121 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:57.722776 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48cab97f-2f9e-4eb4-a11b-7fd35fbc7be0 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:57.723021 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48cab97f-2f9e-4eb4-a11b-7fd35fbc7be0 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:57.723293 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48cab97f-2f9e-4eb4-a11b-7fd35fbc7be0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:57.745707 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-48cab97f-2f9e-4eb4-a11b-7fd35fbc7be0 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:57.746999 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-48cab97f-2f9e-4eb4-a11b-7fd35fbc7be0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:57.806855 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:57.824719 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1166/2297] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:57:57.900830 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-116b1ac9-eca1-4c9f-a36a-a56728a39770 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:57.900830 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-116b1ac9-eca1-4c9f-a36a-a56728a39770 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:57.900830 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-116b1ac9-eca1-4c9f-a36a-a56728a39770 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:57.917922 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-116b1ac9-eca1-4c9f-a36a-a56728a39770 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:57.919129 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-116b1ac9-eca1-4c9f-a36a-a56728a39770 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:57.927390 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1132/2298] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:57:57.931834 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48cab97f-2f9e-4eb4-a11b-7fd35fbc7be0 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:57.942154 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48cab97f-2f9e-4eb4-a11b-7fd35fbc7be0 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:57.946719 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-601d26eb-d22d-4d31-93a5-4b634f6280cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:57.947322 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-601d26eb-d22d-4d31-93a5-4b634f6280cb admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:57.947636 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-601d26eb-d22d-4d31-93a5-4b634f6280cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:57.947897 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-601d26eb-d22d-4d31-93a5-4b634f6280cb admin admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:57.948539 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-601d26eb-d22d-4d31-93a5-4b634f6280cb admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:57:57.948840 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-601d26eb-d22d-4d31-93a5-4b634f6280cb admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:57.950477 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-601d26eb-d22d-4d31-93a5-4b634f6280cb admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:57.965530 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1167/2299] 10.222.0.33 () {58 vars in 1153 bytes} [Sun Oct 9 00:57:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:57:57.969872 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-116b1ac9-eca1-4c9f-a36a-a56728a39770 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:57.979071 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-116b1ac9-eca1-4c9f-a36a-a56728a39770 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:57.983686 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-081ee5a8-0c17-48f0-a158-a9c7540e0b13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:57.984275 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-081ee5a8-0c17-48f0-a158-a9c7540e0b13 admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:57.984550 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-081ee5a8-0c17-48f0-a158-a9c7540e0b13 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:57.984820 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-081ee5a8-0c17-48f0-a158-a9c7540e0b13 admin admin] PATH_INFO: `/v3/projects` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:57.985638 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-081ee5a8-0c17-48f0-a158-a9c7540e0b13 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:57.987515 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-081ee5a8-0c17-48f0-a158-a9c7540e0b13 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:57.993275 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:58.018381 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1133/2300] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 00:57:57 2022] POST /identity/v3/projects => generated 391 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 09 00:57:58.023410 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-601d26eb-d22d-4d31-93a5-4b634f6280cb admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:58.033678 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-601d26eb-d22d-4d31-93a5-4b634f6280cb admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:58.091046 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65d06862-44c1-4aa7-b8d2-4731b0799a55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:58.091691 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d06862-44c1-4aa7-b8d2-4731b0799a55 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:58.092035 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d06862-44c1-4aa7-b8d2-4731b0799a55 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:58.092252 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d06862-44c1-4aa7-b8d2-4731b0799a55 admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:58.092890 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65d06862-44c1-4aa7-b8d2-4731b0799a55 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:58.094836 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65d06862-44c1-4aa7-b8d2-4731b0799a55 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:58.116678 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1168/2301] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:57:58 2022] POST /identity/v3/users => generated 296 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:57:58.123867 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-081ee5a8-0c17-48f0-a158-a9c7540e0b13 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:58.134944 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-081ee5a8-0c17-48f0-a158-a9c7540e0b13 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:58.139667 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-711da5ce-acb2-434c-87b3-f068d2296727 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:58.140265 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-711da5ce-acb2-434c-87b3-f068d2296727 admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:58.140543 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-711da5ce-acb2-434c-87b3-f068d2296727 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:58.140821 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-711da5ce-acb2-434c-87b3-f068d2296727 admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:58.141458 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-711da5ce-acb2-434c-87b3-f068d2296727 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:58.143060 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-711da5ce-acb2-434c-87b3-f068d2296727 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:58.153580 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1134/2302] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:57:58 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 09 00:57:58.158292 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65d06862-44c1-4aa7-b8d2-4731b0799a55 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:58.168422 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65d06862-44c1-4aa7-b8d2-4731b0799a55 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:58.173237 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-482d17ca-80ad-4ef3-93a0-f23f6d46ac63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:58.173822 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-482d17ca-80ad-4ef3-93a0-f23f6d46ac63 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:58.174100 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-482d17ca-80ad-4ef3-93a0-f23f6d46ac63 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:58.174366 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-482d17ca-80ad-4ef3-93a0-f23f6d46ac63 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:58.175053 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-482d17ca-80ad-4ef3-93a0-f23f6d46ac63 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:58.177285 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-482d17ca-80ad-4ef3-93a0-f23f6d46ac63 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:58.188928 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1169/2303] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:57:58 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:57:58.193714 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-711da5ce-acb2-434c-87b3-f068d2296727 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:58.203866 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-711da5ce-acb2-434c-87b3-f068d2296727 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:58.208752 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d13b569-329f-420f-8b1e-e29dd81a41d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:58.209378 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d13b569-329f-420f-8b1e-e29dd81a41d7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:58.209656 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d13b569-329f-420f-8b1e-e29dd81a41d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:58.209927 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d13b569-329f-420f-8b1e-e29dd81a41d7 admin admin] PATH_INFO: `/v3/projects/bb307f6a86b24a31901293578f3d8e2b/users/363330fcf0aa434e987001f0ee3e6bbd/roles/588016933bab436d8b7c3ac76827356d` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:58.224740 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d13b569-329f-420f-8b1e-e29dd81a41d7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bb307f6a86b24a31901293578f3d8e2b, user_id=363330fcf0aa434e987001f0ee3e6bbd, role_id=588016933bab436d8b7c3ac76827356d)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:58.227395 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d13b569-329f-420f-8b1e-e29dd81a41d7 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:58.240647 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1135/2304] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:57:58 2022] PUT /identity/v3/projects/bb307f6a86b24a31901293578f3d8e2b/users/363330fcf0aa434e987001f0ee3e6bbd/roles/588016933bab436d8b7c3ac76827356d => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:58.251712 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c219634e-b9ab-4c9c-928c-73709428d8d4 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:58.251994 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c219634e-b9ab-4c9c-928c-73709428d8d4 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:58.252266 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c219634e-b9ab-4c9c-928c-73709428d8d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:58.280711 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-c219634e-b9ab-4c9c-928c-73709428d8d4 None None] MFA Rules not processed for user `363330fcf0aa434e987001f0ee3e6bbd`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:58.282038 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-c219634e-b9ab-4c9c-928c-73709428d8d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:58.347793 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1170/2305] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:58 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:57:58.358882 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d13b569-329f-420f-8b1e-e29dd81a41d7 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:58.360717 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-1d13b569-329f-420f-8b1e-e29dd81a41d7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:58.426756 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d13b569-329f-420f-8b1e-e29dd81a41d7 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:58.430423 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a094b49-6c27-4eb8-a218-1a2610a58169 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3f48335668a4e8f934ac11ff66cd308', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:58.431073 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a094b49-6c27-4eb8-a218-1a2610a58169 service cinder] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:58.431358 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a094b49-6c27-4eb8-a218-1a2610a58169 service cinder] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:58.431672 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a094b49-6c27-4eb8-a218-1a2610a58169 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:58.439867 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a094b49-6c27-4eb8-a218-1a2610a58169 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:58.441918 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a094b49-6c27-4eb8-a218-1a2610a58169 service cinder] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:58.450653 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1136/2306] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:57:58 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:57:58.664468 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c219634e-b9ab-4c9c-928c-73709428d8d4 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:58.665909 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-c219634e-b9ab-4c9c-928c-73709428d8d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:58.740418 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:58.760245 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c219634e-b9ab-4c9c-928c-73709428d8d4 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:58.765967 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-558691e0-41fa-4de3-bea6-b689906027cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:58.766607 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-558691e0-41fa-4de3-bea6-b689906027cc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:58.766889 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-558691e0-41fa-4de3-bea6-b689906027cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:58.767161 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-558691e0-41fa-4de3-bea6-b689906027cc admin admin] PATH_INFO: `/v3/users/92853af5786743dc82d0223c088b401c` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:58.768333 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-558691e0-41fa-4de3-bea6-b689906027cc admin admin] RBAC: Authorizing `identity:delete_user(user_id=92853af5786743dc82d0223c088b401c)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:58.770393 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-558691e0-41fa-4de3-bea6-b689906027cc admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:58.838973 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-558691e0-41fa-4de3-bea6-b689906027cc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '92853af5786743dc82d0223c088b401c'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:58.850394 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-558691e0-41fa-4de3-bea6-b689906027cc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '92853af5786743dc82d0223c088b401c'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:58.852861 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-558691e0-41fa-4de3-bea6-b689906027cc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '92853af5786743dc82d0223c088b401c'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:58.854182 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-558691e0-41fa-4de3-bea6-b689906027cc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '92853af5786743dc82d0223c088b401c'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:58.864937 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-558691e0-41fa-4de3-bea6-b689906027cc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '92853af5786743dc82d0223c088b401c'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:58.885299 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1171/2307] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 00:57:58 2022] DELETE /identity/v3/users/92853af5786743dc82d0223c088b401c => generated 0 bytes in 221 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:57:58.892459 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7acffb45-971b-4417-bf35-c3c094bdb7a6 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:58.892772 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7acffb45-971b-4417-bf35-c3c094bdb7a6 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:58.893292 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7acffb45-971b-4417-bf35-c3c094bdb7a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:58.914429 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-7acffb45-971b-4417-bf35-c3c094bdb7a6 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:57:58.915679 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-7acffb45-971b-4417-bf35-c3c094bdb7a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:58.976926 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1137/2308] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:57:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 09 00:57:58.986958 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-558691e0-41fa-4de3-bea6-b689906027cc admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:58.989065 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-558691e0-41fa-4de3-bea6-b689906027cc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:59.046469 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-558691e0-41fa-4de3-bea6-b689906027cc admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:59.050627 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aeaa624-1f42-4697-892b-2e000db2ffc5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:59.051225 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aeaa624-1f42-4697-892b-2e000db2ffc5 service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:59.051504 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aeaa624-1f42-4697-892b-2e000db2ffc5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:59.051813 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aeaa624-1f42-4697-892b-2e000db2ffc5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:59.060698 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aeaa624-1f42-4697-892b-2e000db2ffc5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:59.062119 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aeaa624-1f42-4697-892b-2e000db2ffc5 service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:59.071923 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1172/2309] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:57:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:57:59.349151 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7acffb45-971b-4417-bf35-c3c094bdb7a6 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:59.350622 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-7acffb45-971b-4417-bf35-c3c094bdb7a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:57:59.363258 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7acffb45-971b-4417-bf35-c3c094bdb7a6 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:59.368219 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-236d7be8-4a28-4d4e-a646-7519b4fe310c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:59.368859 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-236d7be8-4a28-4d4e-a646-7519b4fe310c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:59.369918 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-236d7be8-4a28-4d4e-a646-7519b4fe310c admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:59.369918 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-236d7be8-4a28-4d4e-a646-7519b4fe310c admin admin] PATH_INFO: `/v3/projects/12ae8ab3a3a64ff6887430aab9669232` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:59.370896 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-236d7be8-4a28-4d4e-a646-7519b4fe310c admin admin] RBAC: Authorizing `identity:delete_project(project_id=12ae8ab3a3a64ff6887430aab9669232)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:59.372824 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-236d7be8-4a28-4d4e-a646-7519b4fe310c admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:59.421259 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=94,keystone:INSERT=2,keystone:DELETE=4 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:59.423078 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-236d7be8-4a28-4d4e-a646-7519b4fe310c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '12ae8ab3a3a64ff6887430aab9669232', 'request_id': 'req-236d7be8-4a28-4d4e-a646-7519b4fe310c'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:59.432995 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-236d7be8-4a28-4d4e-a646-7519b4fe310c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '12ae8ab3a3a64ff6887430aab9669232', 'request_id': 'req-236d7be8-4a28-4d4e-a646-7519b4fe310c'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:59.441905 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-236d7be8-4a28-4d4e-a646-7519b4fe310c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '12ae8ab3a3a64ff6887430aab9669232', 'request_id': 'req-236d7be8-4a28-4d4e-a646-7519b4fe310c'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:59.470718 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-236d7be8-4a28-4d4e-a646-7519b4fe310c admin admin] The token cache is being invalidate because project 12ae8ab3a3a64ff6887430aab9669232 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81453) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 00:57:59.471291 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-236d7be8-4a28-4d4e-a646-7519b4fe310c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:59.472629 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-236d7be8-4a28-4d4e-a646-7519b4fe310c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:59.474670 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1138/2310] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 00:57:59 2022] DELETE /identity/v3/projects/12ae8ab3a3a64ff6887430aab9669232 => generated 0 bytes in 126 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 09 00:57:59.478742 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aeaa624-1f42-4697-892b-2e000db2ffc5 service neutron] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:59.489912 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aeaa624-1f42-4697-892b-2e000db2ffc5 service neutron] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:57:59.556895 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d8598b0-0ea8-40df-bf25-e8e920169f7f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:57:59.557525 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d8598b0-0ea8-40df-bf25-e8e920169f7f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:57:59.557806 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d8598b0-0ea8-40df-bf25-e8e920169f7f admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:57:59.558080 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d8598b0-0ea8-40df-bf25-e8e920169f7f admin admin] PATH_INFO: `/v3/users/a1cf02e41cf94286bf6e0d48ca343d47` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:57:59.559193 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d8598b0-0ea8-40df-bf25-e8e920169f7f admin admin] RBAC: Authorizing `identity:delete_user(user_id=a1cf02e41cf94286bf6e0d48ca343d47)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:57:59.561615 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d8598b0-0ea8-40df-bf25-e8e920169f7f admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:57:59.573257 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:57:59.615626 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-0d8598b0-0ea8-40df-bf25-e8e920169f7f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a1cf02e41cf94286bf6e0d48ca343d47'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:59.623114 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-0d8598b0-0ea8-40df-bf25-e8e920169f7f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a1cf02e41cf94286bf6e0d48ca343d47'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:59.623829 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-0d8598b0-0ea8-40df-bf25-e8e920169f7f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a1cf02e41cf94286bf6e0d48ca343d47'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:59.624436 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-0d8598b0-0ea8-40df-bf25-e8e920169f7f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a1cf02e41cf94286bf6e0d48ca343d47'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:59.632028 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-0d8598b0-0ea8-40df-bf25-e8e920169f7f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a1cf02e41cf94286bf6e0d48ca343d47'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:57:59.651095 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1173/2311] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 00:57:59 2022] DELETE /identity/v3/users/a1cf02e41cf94286bf6e0d48ca343d47 => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 09 00:57:59.978572 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-236d7be8-4a28-4d4e-a646-7519b4fe310c admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:57:59.980085 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-236d7be8-4a28-4d4e-a646-7519b4fe310c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:58:00.016381 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-236d7be8-4a28-4d4e-a646-7519b4fe310c admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:00.131755 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5db902e-f857-4b14-bc54-ec02c3fe9d52 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:00.131755 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5db902e-f857-4b14-bc54-ec02c3fe9d52 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:00.131755 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5db902e-f857-4b14-bc54-ec02c3fe9d52 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:00.131755 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5db902e-f857-4b14-bc54-ec02c3fe9d52 admin admin] PATH_INFO: `/v3/projects/211862bbf0dd4e7092c1c33f6321459c` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:00.132213 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5db902e-f857-4b14-bc54-ec02c3fe9d52 admin admin] RBAC: Authorizing `identity:delete_project(project_id=211862bbf0dd4e7092c1c33f6321459c)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:00.132213 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5db902e-f857-4b14-bc54-ec02c3fe9d52 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:00.179152 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-a5db902e-f857-4b14-bc54-ec02c3fe9d52 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '211862bbf0dd4e7092c1c33f6321459c', 'request_id': 'req-a5db902e-f857-4b14-bc54-ec02c3fe9d52'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:00.184787 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-a5db902e-f857-4b14-bc54-ec02c3fe9d52 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '211862bbf0dd4e7092c1c33f6321459c', 'request_id': 'req-a5db902e-f857-4b14-bc54-ec02c3fe9d52'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:00.191390 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-a5db902e-f857-4b14-bc54-ec02c3fe9d52 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '211862bbf0dd4e7092c1c33f6321459c', 'request_id': 'req-a5db902e-f857-4b14-bc54-ec02c3fe9d52'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:00.203852 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-a5db902e-f857-4b14-bc54-ec02c3fe9d52 admin admin] The token cache is being invalidate because project 211862bbf0dd4e7092c1c33f6321459c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81453) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 00:58:00.204234 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-a5db902e-f857-4b14-bc54-ec02c3fe9d52 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:00.204852 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-a5db902e-f857-4b14-bc54-ec02c3fe9d52 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:00.207024 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1139/2312] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 00:57:59 2022] DELETE /identity/v3/projects/211862bbf0dd4e7092c1c33f6321459c => generated 0 bytes in 230 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 09 00:58:00.212470 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09db92c4-6a4b-4267-8b92-c9b0dc812960 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:00.213020 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09db92c4-6a4b-4267-8b92-c9b0dc812960 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:00.213020 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09db92c4-6a4b-4267-8b92-c9b0dc812960 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:00.232502 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-09db92c4-6a4b-4267-8b92-c9b0dc812960 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:58:00.233696 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-09db92c4-6a4b-4267-8b92-c9b0dc812960 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:58:00.300685 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1174/2313] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:58:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 09 00:58:00.308642 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84fcd97d-faa2-40af-b87b-3f0c91751876 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:00.308927 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84fcd97d-faa2-40af-b87b-3f0c91751876 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:00.309204 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84fcd97d-faa2-40af-b87b-3f0c91751876 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:00.329287 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-84fcd97d-faa2-40af-b87b-3f0c91751876 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:58:00.330510 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-84fcd97d-faa2-40af-b87b-3f0c91751876 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:58:00.339981 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1140/2314] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:58:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:58:00.344381 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09db92c4-6a4b-4267-8b92-c9b0dc812960 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:00.354310 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09db92c4-6a4b-4267-8b92-c9b0dc812960 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:00.358916 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-616a44fa-0fc5-48ae-aeca-ea7898c19b1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:00.359509 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-616a44fa-0fc5-48ae-aeca-ea7898c19b1c admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:00.359827 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-616a44fa-0fc5-48ae-aeca-ea7898c19b1c admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:00.360103 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-616a44fa-0fc5-48ae-aeca-ea7898c19b1c admin admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:00.360746 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-616a44fa-0fc5-48ae-aeca-ea7898c19b1c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:58:00.361055 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-616a44fa-0fc5-48ae-aeca-ea7898c19b1c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:00.362646 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-616a44fa-0fc5-48ae-aeca-ea7898c19b1c admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:00.373982 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1175/2315] 10.222.0.33 () {58 vars in 1153 bytes} [Sun Oct 9 00:58:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:58:00.378908 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84fcd97d-faa2-40af-b87b-3f0c91751876 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:00.388063 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84fcd97d-faa2-40af-b87b-3f0c91751876 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:00.392877 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48b2af7e-fb3c-4355-919f-72524e5d52af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:00.393479 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48b2af7e-fb3c-4355-919f-72524e5d52af admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:00.394399 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48b2af7e-fb3c-4355-919f-72524e5d52af admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:00.394679 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48b2af7e-fb3c-4355-919f-72524e5d52af admin admin] PATH_INFO: `/v3/projects` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:00.395317 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48b2af7e-fb3c-4355-919f-72524e5d52af admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:00.397287 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48b2af7e-fb3c-4355-919f-72524e5d52af admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:00.425914 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1141/2316] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 00:58:00 2022] POST /identity/v3/projects => generated 371 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:58:00.429479 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-616a44fa-0fc5-48ae-aeca-ea7898c19b1c admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:00.439810 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-616a44fa-0fc5-48ae-aeca-ea7898c19b1c admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:00.504315 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6763150b-f81c-4ba9-b695-bed333fe1d17 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:00.504928 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6763150b-f81c-4ba9-b695-bed333fe1d17 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:00.505208 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6763150b-f81c-4ba9-b695-bed333fe1d17 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:00.505479 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6763150b-f81c-4ba9-b695-bed333fe1d17 admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:00.506123 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6763150b-f81c-4ba9-b695-bed333fe1d17 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:00.508312 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6763150b-f81c-4ba9-b695-bed333fe1d17 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:00.534432 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1176/2317] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:58:00 2022] POST /identity/v3/users => generated 286 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:58:00.537440 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48b2af7e-fb3c-4355-919f-72524e5d52af admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:00.552877 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48b2af7e-fb3c-4355-919f-72524e5d52af admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:00.564501 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8e9cae1-666c-4e37-90e5-b336497f582a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:00.565124 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e9cae1-666c-4e37-90e5-b336497f582a admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:00.565404 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e9cae1-666c-4e37-90e5-b336497f582a admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:00.566283 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e9cae1-666c-4e37-90e5-b336497f582a admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:00.566957 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8e9cae1-666c-4e37-90e5-b336497f582a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:00.568722 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8e9cae1-666c-4e37-90e5-b336497f582a admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:00.579245 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1142/2318] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:58:00 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:58:00.584890 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6763150b-f81c-4ba9-b695-bed333fe1d17 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:00.591127 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:DELETE=10,keystone:SELECT=86,keystone:INSERT=4 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:58:00.595895 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6763150b-f81c-4ba9-b695-bed333fe1d17 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:00.605442 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54667841-4a25-40ec-9dd1-369e5a21f8a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:00.606726 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54667841-4a25-40ec-9dd1-369e5a21f8a1 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:00.607849 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54667841-4a25-40ec-9dd1-369e5a21f8a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:00.608158 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54667841-4a25-40ec-9dd1-369e5a21f8a1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:00.608982 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54667841-4a25-40ec-9dd1-369e5a21f8a1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:00.610823 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54667841-4a25-40ec-9dd1-369e5a21f8a1 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:00.624233 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1177/2319] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:58:00 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:58:00.629022 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8e9cae1-666c-4e37-90e5-b336497f582a admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:00.638894 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8e9cae1-666c-4e37-90e5-b336497f582a admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:00.643669 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fabd90c-2e19-476d-b4b8-aef24809eb14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:00.644332 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fabd90c-2e19-476d-b4b8-aef24809eb14 admin admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:00.644614 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fabd90c-2e19-476d-b4b8-aef24809eb14 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:00.644895 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fabd90c-2e19-476d-b4b8-aef24809eb14 admin admin] PATH_INFO: `/v3/projects/23d6fc625f854ae4945b67513046d2ed/users/8ad78c51d6f640e0ab972a8504fe2e39/roles/588016933bab436d8b7c3ac76827356d` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:00.660330 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fabd90c-2e19-476d-b4b8-aef24809eb14 admin admin] RBAC: Authorizing `identity:create_grant(project_id=23d6fc625f854ae4945b67513046d2ed, user_id=8ad78c51d6f640e0ab972a8504fe2e39, role_id=588016933bab436d8b7c3ac76827356d)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:00.662996 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fabd90c-2e19-476d-b4b8-aef24809eb14 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:00.676378 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1143/2320] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:58:00 2022] PUT /identity/v3/projects/23d6fc625f854ae4945b67513046d2ed/users/8ad78c51d6f640e0ab972a8504fe2e39/roles/588016933bab436d8b7c3ac76827356d => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:58:00.681276 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254fc99c-72fb-43f2-9b66-0c7e8e27b401 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:00.681563 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254fc99c-72fb-43f2-9b66-0c7e8e27b401 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:00.681837 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254fc99c-72fb-43f2-9b66-0c7e8e27b401 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:00.700983 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-254fc99c-72fb-43f2-9b66-0c7e8e27b401 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:58:00.702189 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-254fc99c-72fb-43f2-9b66-0c7e8e27b401 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:58:00.762116 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1178/2321] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:58:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:58:00.771736 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fabd90c-2e19-476d-b4b8-aef24809eb14 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:00.773598 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-4fabd90c-2e19-476d-b4b8-aef24809eb14 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:58:00.778975 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:58:00.832985 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fabd90c-2e19-476d-b4b8-aef24809eb14 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:00.836762 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4302d65-93a0-4fd0-8465-d3db5d16963e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:00.837402 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4302d65-93a0-4fd0-8465-d3db5d16963e service neutron] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:00.837683 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4302d65-93a0-4fd0-8465-d3db5d16963e service neutron] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:00.837958 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4302d65-93a0-4fd0-8465-d3db5d16963e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:00.848049 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4302d65-93a0-4fd0-8465-d3db5d16963e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:00.849657 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4302d65-93a0-4fd0-8465-d3db5d16963e service neutron] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:00.860230 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1144/2322] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:58:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:58:01.773384 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5a1b8f-0725-49d8-a34e-6f547cfbac33 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:01.773978 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5a1b8f-0725-49d8-a34e-6f547cfbac33 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:01.773978 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5a1b8f-0725-49d8-a34e-6f547cfbac33 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:01.805221 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-fe5a1b8f-0725-49d8-a34e-6f547cfbac33 None None] MFA Rules not processed for user `8ad78c51d6f640e0ab972a8504fe2e39`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:58:01.806457 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-fe5a1b8f-0725-49d8-a34e-6f547cfbac33 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:58:01.875079 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1179/2323] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:58:01 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:58:01.879881 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4302d65-93a0-4fd0-8465-d3db5d16963e service neutron] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:01.889898 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4302d65-93a0-4fd0-8465-d3db5d16963e service neutron] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:01.895044 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6506ee2-ba00-4740-83bd-7521026e4484 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:01.895676 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6506ee2-ba00-4740-83bd-7521026e4484 admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:01.895949 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6506ee2-ba00-4740-83bd-7521026e4484 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:01.896224 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6506ee2-ba00-4740-83bd-7521026e4484 admin admin] PATH_INFO: `/v3/projects` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:01.896860 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6506ee2-ba00-4740-83bd-7521026e4484 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:01.899263 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6506ee2-ba00-4740-83bd-7521026e4484 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:01.934828 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1145/2324] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 00:58:01 2022] POST /identity/v3/projects => generated 373 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:58:01.938906 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe5a1b8f-0725-49d8-a34e-6f547cfbac33 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:01.950553 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe5a1b8f-0725-49d8-a34e-6f547cfbac33 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:02.015281 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bfe2171-6935-4a3b-9651-908cd595e344 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:02.015958 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bfe2171-6935-4a3b-9651-908cd595e344 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:02.016240 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bfe2171-6935-4a3b-9651-908cd595e344 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:02.016517 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bfe2171-6935-4a3b-9651-908cd595e344 admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:02.017169 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bfe2171-6935-4a3b-9651-908cd595e344 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:02.019838 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bfe2171-6935-4a3b-9651-908cd595e344 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:02.046720 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1180/2325] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:58:01 2022] POST /identity/v3/users => generated 293 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:58:02.051307 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6506ee2-ba00-4740-83bd-7521026e4484 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:02.061565 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6506ee2-ba00-4740-83bd-7521026e4484 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:02.067035 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaa70906-4cf9-4537-af0e-9c0e94305722 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:02.067648 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaa70906-4cf9-4537-af0e-9c0e94305722 admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:02.067932 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaa70906-4cf9-4537-af0e-9c0e94305722 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:02.068207 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaa70906-4cf9-4537-af0e-9c0e94305722 admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:02.068838 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaa70906-4cf9-4537-af0e-9c0e94305722 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:02.070424 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaa70906-4cf9-4537-af0e-9c0e94305722 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:02.085815 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1146/2326] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:58:02 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:58:02.089420 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bfe2171-6935-4a3b-9651-908cd595e344 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:02.099493 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bfe2171-6935-4a3b-9651-908cd595e344 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:02.104285 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcdec6e0-2f5c-48b8-a7e3-c32081b13864 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:02.104980 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcdec6e0-2f5c-48b8-a7e3-c32081b13864 admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:02.105206 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcdec6e0-2f5c-48b8-a7e3-c32081b13864 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:02.105476 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcdec6e0-2f5c-48b8-a7e3-c32081b13864 admin admin] PATH_INFO: `/v3/projects/e1ccdbaa29bd464ba64ad77ff7a00244/users/a7b9aef56550472fb19a34543853b6ec/roles/092036ee46ac427baf4d56468575f697` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:02.117295 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:58:02.121840 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcdec6e0-2f5c-48b8-a7e3-c32081b13864 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e1ccdbaa29bd464ba64ad77ff7a00244, user_id=a7b9aef56550472fb19a34543853b6ec, role_id=092036ee46ac427baf4d56468575f697)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:02.125012 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcdec6e0-2f5c-48b8-a7e3-c32081b13864 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:02.137913 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1181/2327] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:58:02 2022] PUT /identity/v3/projects/e1ccdbaa29bd464ba64ad77ff7a00244/users/a7b9aef56550472fb19a34543853b6ec/roles/092036ee46ac427baf4d56468575f697 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:58:02.142310 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaa70906-4cf9-4537-af0e-9c0e94305722 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:02.153612 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaa70906-4cf9-4537-af0e-9c0e94305722 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:02.212498 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f02db50f-a469-41fa-9400-083fafbaf443 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:02.213094 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f02db50f-a469-41fa-9400-083fafbaf443 admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:02.213374 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f02db50f-a469-41fa-9400-083fafbaf443 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:02.213642 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f02db50f-a469-41fa-9400-083fafbaf443 admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:02.214261 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f02db50f-a469-41fa-9400-083fafbaf443 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:02.215857 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f02db50f-a469-41fa-9400-083fafbaf443 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:02.225201 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1147/2328] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:58:02 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:58:02.229356 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcdec6e0-2f5c-48b8-a7e3-c32081b13864 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:02.240024 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcdec6e0-2f5c-48b8-a7e3-c32081b13864 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:02.244557 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7ddcfca-81a7-4e69-82cb-fd671c5d7fc3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:02.245191 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7ddcfca-81a7-4e69-82cb-fd671c5d7fc3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:02.245467 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7ddcfca-81a7-4e69-82cb-fd671c5d7fc3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:02.245739 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7ddcfca-81a7-4e69-82cb-fd671c5d7fc3 admin admin] PATH_INFO: `/v3/projects/e1ccdbaa29bd464ba64ad77ff7a00244/users/a7b9aef56550472fb19a34543853b6ec/roles/092036ee46ac427baf4d56468575f697` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:02.247620 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7ddcfca-81a7-4e69-82cb-fd671c5d7fc3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e1ccdbaa29bd464ba64ad77ff7a00244, user_id=a7b9aef56550472fb19a34543853b6ec, role_id=092036ee46ac427baf4d56468575f697)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:02.250244 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7ddcfca-81a7-4e69-82cb-fd671c5d7fc3 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:02.261566 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1182/2329] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:58:02 2022] PUT /identity/v3/projects/e1ccdbaa29bd464ba64ad77ff7a00244/users/a7b9aef56550472fb19a34543853b6ec/roles/092036ee46ac427baf4d56468575f697 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:58:03.015229 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1a3bd30-d643-4604-a8c4-1f82cec9afd0 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:03.015825 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1a3bd30-d643-4604-a8c4-1f82cec9afd0 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:03.015825 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1a3bd30-d643-4604-a8c4-1f82cec9afd0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:03.047262 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-b1a3bd30-d643-4604-a8c4-1f82cec9afd0 None None] MFA Rules not processed for user `a7b9aef56550472fb19a34543853b6ec`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:58:03.048546 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-b1a3bd30-d643-4604-a8c4-1f82cec9afd0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:58:03.121522 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:58:03.136882 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1148/2330] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:58:03 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:58:03.148666 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7ddcfca-81a7-4e69-82cb-fd671c5d7fc3 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:03.150883 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-b7ddcfca-81a7-4e69-82cb-fd671c5d7fc3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:58:03.212691 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7ddcfca-81a7-4e69-82cb-fd671c5d7fc3 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:03.219333 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-116a372c-2c35-4403-9c07-81848e1f85f9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3f48335668a4e8f934ac11ff66cd308', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:03.220085 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-116a372c-2c35-4403-9c07-81848e1f85f9 service cinder] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:03.220366 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-116a372c-2c35-4403-9c07-81848e1f85f9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:03.220653 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-116a372c-2c35-4403-9c07-81848e1f85f9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:03.284285 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-116a372c-2c35-4403-9c07-81848e1f85f9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:03.285712 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-116a372c-2c35-4403-9c07-81848e1f85f9 service cinder] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:03.294334 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1183/2331] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:58:03 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 146 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:58:04.657302 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1a3bd30-d643-4604-a8c4-1f82cec9afd0 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:04.658707 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-b1a3bd30-d643-4604-a8c4-1f82cec9afd0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:58:04.670394 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1a3bd30-d643-4604-a8c4-1f82cec9afd0 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:04.673873 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6bafd6a-ddf2-4f4a-b283-179fbbc87eac service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3f48335668a4e8f934ac11ff66cd308', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:04.674480 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6bafd6a-ddf2-4f4a-b283-179fbbc87eac service cinder] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:04.674824 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6bafd6a-ddf2-4f4a-b283-179fbbc87eac service cinder] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:04.675006 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6bafd6a-ddf2-4f4a-b283-179fbbc87eac service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:04.684318 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6bafd6a-ddf2-4f4a-b283-179fbbc87eac service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:04.685821 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6bafd6a-ddf2-4f4a-b283-179fbbc87eac service cinder] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:04.698022 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1149/2332] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:58:04 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:58:13.290436 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=57,keystone:INSERT=2 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:58:14.694289 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=9 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:58:18.053230 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-116a372c-2c35-4403-9c07-81848e1f85f9 service cinder] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:18.055277 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-116a372c-2c35-4403-9c07-81848e1f85f9 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:58:18.132505 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-116a372c-2c35-4403-9c07-81848e1f85f9 service cinder] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:18.137745 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6bca5b6-84aa-4d1c-b8e8-87ce47aab505 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:18.138367 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6bca5b6-84aa-4d1c-b8e8-87ce47aab505 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:18.138642 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6bca5b6-84aa-4d1c-b8e8-87ce47aab505 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:18.138911 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6bca5b6-84aa-4d1c-b8e8-87ce47aab505 admin admin] PATH_INFO: `/v3/users/363330fcf0aa434e987001f0ee3e6bbd` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:18.140058 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6bca5b6-84aa-4d1c-b8e8-87ce47aab505 admin admin] RBAC: Authorizing `identity:delete_user(user_id=363330fcf0aa434e987001f0ee3e6bbd)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:18.142635 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6bca5b6-84aa-4d1c-b8e8-87ce47aab505 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:18.194237 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-e6bca5b6-84aa-4d1c-b8e8-87ce47aab505 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '363330fcf0aa434e987001f0ee3e6bbd'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:18.202406 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-e6bca5b6-84aa-4d1c-b8e8-87ce47aab505 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '363330fcf0aa434e987001f0ee3e6bbd'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:18.203167 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-e6bca5b6-84aa-4d1c-b8e8-87ce47aab505 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '363330fcf0aa434e987001f0ee3e6bbd'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:18.203940 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-e6bca5b6-84aa-4d1c-b8e8-87ce47aab505 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '363330fcf0aa434e987001f0ee3e6bbd'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:18.214297 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-e6bca5b6-84aa-4d1c-b8e8-87ce47aab505 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '363330fcf0aa434e987001f0ee3e6bbd'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:18.240395 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1184/2333] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 00:58:18 2022] DELETE /identity/v3/users/363330fcf0aa434e987001f0ee3e6bbd => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:58:18.244802 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ee699e5-ad55-4b1b-b55d-b109f6a4dd12 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:18.245309 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ee699e5-ad55-4b1b-b55d-b109f6a4dd12 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:18.245409 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ee699e5-ad55-4b1b-b55d-b109f6a4dd12 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:18.268364 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-1ee699e5-ad55-4b1b-b55d-b109f6a4dd12 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:58:18.269689 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-1ee699e5-ad55-4b1b-b55d-b109f6a4dd12 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:58:18.333526 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1150/2334] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:58:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:58:18.343101 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6bca5b6-84aa-4d1c-b8e8-87ce47aab505 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:18.344552 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-e6bca5b6-84aa-4d1c-b8e8-87ce47aab505 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:58:18.407731 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6bca5b6-84aa-4d1c-b8e8-87ce47aab505 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:18.412029 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-766e6595-4a32-45b4-ad49-bb1b1b048bd9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:18.412625 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-766e6595-4a32-45b4-ad49-bb1b1b048bd9 service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:18.412906 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-766e6595-4a32-45b4-ad49-bb1b1b048bd9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:18.413180 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-766e6595-4a32-45b4-ad49-bb1b1b048bd9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:18.422038 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-766e6595-4a32-45b4-ad49-bb1b1b048bd9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:18.423460 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-766e6595-4a32-45b4-ad49-bb1b1b048bd9 service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:18.433358 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1185/2335] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:58:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:58:18.710113 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ee699e5-ad55-4b1b-b55d-b109f6a4dd12 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:18.711555 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-1ee699e5-ad55-4b1b-b55d-b109f6a4dd12 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:58:18.724210 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ee699e5-ad55-4b1b-b55d-b109f6a4dd12 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:18.728877 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53b7d4f0-294d-48cc-985c-009d935cdf4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:18.729699 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53b7d4f0-294d-48cc-985c-009d935cdf4d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:18.729985 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53b7d4f0-294d-48cc-985c-009d935cdf4d admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:18.730269 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53b7d4f0-294d-48cc-985c-009d935cdf4d admin admin] PATH_INFO: `/v3/projects/bb307f6a86b24a31901293578f3d8e2b` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:18.731399 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53b7d4f0-294d-48cc-985c-009d935cdf4d admin admin] RBAC: Authorizing `identity:delete_project(project_id=bb307f6a86b24a31901293578f3d8e2b)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:18.733352 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53b7d4f0-294d-48cc-985c-009d935cdf4d admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:18.790789 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-53b7d4f0-294d-48cc-985c-009d935cdf4d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bb307f6a86b24a31901293578f3d8e2b', 'request_id': 'req-53b7d4f0-294d-48cc-985c-009d935cdf4d'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:18.796695 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-53b7d4f0-294d-48cc-985c-009d935cdf4d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bb307f6a86b24a31901293578f3d8e2b', 'request_id': 'req-53b7d4f0-294d-48cc-985c-009d935cdf4d'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:18.803153 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-53b7d4f0-294d-48cc-985c-009d935cdf4d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bb307f6a86b24a31901293578f3d8e2b', 'request_id': 'req-53b7d4f0-294d-48cc-985c-009d935cdf4d'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:18.815336 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-53b7d4f0-294d-48cc-985c-009d935cdf4d admin admin] The token cache is being invalidate because project bb307f6a86b24a31901293578f3d8e2b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81453) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 00:58:18.815663 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-53b7d4f0-294d-48cc-985c-009d935cdf4d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:18.816810 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-53b7d4f0-294d-48cc-985c-009d935cdf4d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:18.818296 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1151/2336] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 00:58:18 2022] DELETE /identity/v3/projects/bb307f6a86b24a31901293578f3d8e2b => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:58:22.032541 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eaada0c-fc25-4846-9f2a-8c098c60ad87 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:22.033112 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eaada0c-fc25-4846-9f2a-8c098c60ad87 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:22.033112 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eaada0c-fc25-4846-9f2a-8c098c60ad87 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:22.052996 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-4eaada0c-fc25-4846-9f2a-8c098c60ad87 None None] MFA Rules not processed for user `b027fb8adadc488e975c4541e7ee6626`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:58:22.054173 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-4eaada0c-fc25-4846-9f2a-8c098c60ad87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:58:22.075644 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:58:22.119718 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1186/2337] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:58:22 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:58:22.755308 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d39240b-aee6-45ce-9c62-fa6f6ca11201 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:22.755843 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d39240b-aee6-45ce-9c62-fa6f6ca11201 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:22.755934 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d39240b-aee6-45ce-9c62-fa6f6ca11201 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:22.757372 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1152/2338] 10.222.0.33 () {54 vars in 921 bytes} [Sun Oct 9 00:58:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 00:58:22.769152 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33fe8bdf-02f8-4592-8a77-d1ebfa560cf6 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:22.769442 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33fe8bdf-02f8-4592-8a77-d1ebfa560cf6 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:22.769718 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33fe8bdf-02f8-4592-8a77-d1ebfa560cf6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:22.794613 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-33fe8bdf-02f8-4592-8a77-d1ebfa560cf6 None None] MFA Rules not processed for user `d25e240cb93643ed8122ccac9586936f`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:58:22.796043 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-33fe8bdf-02f8-4592-8a77-d1ebfa560cf6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:58:22.889667 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1187/2339] 10.222.0.33 () {58 vars in 1018 bytes} [Sun Oct 9 00:58:22 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 09 00:58:22.903247 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d39240b-aee6-45ce-9c62-fa6f6ca11201 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:22.905130 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-7d39240b-aee6-45ce-9c62-fa6f6ca11201 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:58:22.923624 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d39240b-aee6-45ce-9c62-fa6f6ca11201 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:22.928727 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8350ff73-66ba-41cd-a986-ef5e7128f287 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd25e240cb93643ed8122ccac9586936f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:22.929411 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8350ff73-66ba-41cd-a986-ef5e7128f287 service placement] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:22.929750 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8350ff73-66ba-41cd-a986-ef5e7128f287 service placement] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:22.930225 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8350ff73-66ba-41cd-a986-ef5e7128f287 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:22.939878 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8350ff73-66ba-41cd-a986-ef5e7128f287 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:22.941809 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8350ff73-66ba-41cd-a986-ef5e7128f287 service placement] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:22.962641 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1153/2340] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:58:22 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 59 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:58:23.442088 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33fe8bdf-02f8-4592-8a77-d1ebfa560cf6 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:23.443631 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-33fe8bdf-02f8-4592-8a77-d1ebfa560cf6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:58:23.522038 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33fe8bdf-02f8-4592-8a77-d1ebfa560cf6 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:23.526651 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c386f31-ea59-4871-875a-df37cfa600a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:23.527281 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c386f31-ea59-4871-875a-df37cfa600a7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:23.527572 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c386f31-ea59-4871-875a-df37cfa600a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:23.527878 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c386f31-ea59-4871-875a-df37cfa600a7 admin admin] PATH_INFO: `/v3/users/8ad78c51d6f640e0ab972a8504fe2e39` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:23.529031 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c386f31-ea59-4871-875a-df37cfa600a7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8ad78c51d6f640e0ab972a8504fe2e39)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:23.531415 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c386f31-ea59-4871-875a-df37cfa600a7 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:23.598396 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-2c386f31-ea59-4871-875a-df37cfa600a7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8ad78c51d6f640e0ab972a8504fe2e39'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:23.599048 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-2c386f31-ea59-4871-875a-df37cfa600a7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8ad78c51d6f640e0ab972a8504fe2e39'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:23.599839 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-2c386f31-ea59-4871-875a-df37cfa600a7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8ad78c51d6f640e0ab972a8504fe2e39'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:23.600701 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-2c386f31-ea59-4871-875a-df37cfa600a7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8ad78c51d6f640e0ab972a8504fe2e39'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:23.609158 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-2c386f31-ea59-4871-875a-df37cfa600a7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8ad78c51d6f640e0ab972a8504fe2e39'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:23.611292 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:58:23.631547 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1188/2341] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 00:58:23 2022] DELETE /identity/v3/users/8ad78c51d6f640e0ab972a8504fe2e39 => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:58:23.827870 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8350ff73-66ba-41cd-a986-ef5e7128f287 service placement] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:23.847661 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8350ff73-66ba-41cd-a986-ef5e7128f287 service placement] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:23.918448 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25beace5-9ea7-46a0-84fc-57af8b60cce1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:23.919076 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25beace5-9ea7-46a0-84fc-57af8b60cce1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:23.919354 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25beace5-9ea7-46a0-84fc-57af8b60cce1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:23.919654 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25beace5-9ea7-46a0-84fc-57af8b60cce1 admin admin] PATH_INFO: `/v3/projects/23d6fc625f854ae4945b67513046d2ed` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:23.920777 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25beace5-9ea7-46a0-84fc-57af8b60cce1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=23d6fc625f854ae4945b67513046d2ed)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:23.922668 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25beace5-9ea7-46a0-84fc-57af8b60cce1 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:23.935158 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:58:23.977308 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-25beace5-9ea7-46a0-84fc-57af8b60cce1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '23d6fc625f854ae4945b67513046d2ed', 'request_id': 'req-25beace5-9ea7-46a0-84fc-57af8b60cce1'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:23.982951 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-25beace5-9ea7-46a0-84fc-57af8b60cce1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '23d6fc625f854ae4945b67513046d2ed', 'request_id': 'req-25beace5-9ea7-46a0-84fc-57af8b60cce1'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:23.991274 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-25beace5-9ea7-46a0-84fc-57af8b60cce1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '23d6fc625f854ae4945b67513046d2ed', 'request_id': 'req-25beace5-9ea7-46a0-84fc-57af8b60cce1'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:24.004482 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-25beace5-9ea7-46a0-84fc-57af8b60cce1 admin admin] The token cache is being invalidate because project 23d6fc625f854ae4945b67513046d2ed was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81453) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 00:58:24.004841 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-25beace5-9ea7-46a0-84fc-57af8b60cce1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:24.005467 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-25beace5-9ea7-46a0-84fc-57af8b60cce1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:24.007049 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1154/2342] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 00:58:23 2022] DELETE /identity/v3/projects/23d6fc625f854ae4945b67513046d2ed => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:58:24.010989 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c386f31-ea59-4871-875a-df37cfa600a7 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:24.012423 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-2c386f31-ea59-4871-875a-df37cfa600a7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:58:24.025329 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c386f31-ea59-4871-875a-df37cfa600a7 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:24.139752 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79d41c7d-62ae-429a-aa2e-678c3f56bb8c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:24.140392 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d41c7d-62ae-429a-aa2e-678c3f56bb8c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:24.140670 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d41c7d-62ae-429a-aa2e-678c3f56bb8c admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:24.140950 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d41c7d-62ae-429a-aa2e-678c3f56bb8c admin admin] PATH_INFO: `/v3/users/a7b9aef56550472fb19a34543853b6ec` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:24.142076 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79d41c7d-62ae-429a-aa2e-678c3f56bb8c admin admin] RBAC: Authorizing `identity:delete_user(user_id=a7b9aef56550472fb19a34543853b6ec)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:24.145206 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79d41c7d-62ae-429a-aa2e-678c3f56bb8c admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:24.202603 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-79d41c7d-62ae-429a-aa2e-678c3f56bb8c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a7b9aef56550472fb19a34543853b6ec'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:24.210199 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-79d41c7d-62ae-429a-aa2e-678c3f56bb8c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a7b9aef56550472fb19a34543853b6ec'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:24.210959 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-79d41c7d-62ae-429a-aa2e-678c3f56bb8c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a7b9aef56550472fb19a34543853b6ec'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:24.211568 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-79d41c7d-62ae-429a-aa2e-678c3f56bb8c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a7b9aef56550472fb19a34543853b6ec'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:24.219837 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-79d41c7d-62ae-429a-aa2e-678c3f56bb8c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a7b9aef56550472fb19a34543853b6ec'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:24.242672 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1189/2343] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 00:58:24 2022] DELETE /identity/v3/users/a7b9aef56550472fb19a34543853b6ec => generated 0 bytes in 229 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:58:24.519989 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25beace5-9ea7-46a0-84fc-57af8b60cce1 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:24.521512 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-25beace5-9ea7-46a0-84fc-57af8b60cce1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:58:24.538409 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25beace5-9ea7-46a0-84fc-57af8b60cce1 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:24.598775 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50fbb6a7-bf38-4c1e-ba6d-43923782df3a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:24.599411 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50fbb6a7-bf38-4c1e-ba6d-43923782df3a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:24.599721 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50fbb6a7-bf38-4c1e-ba6d-43923782df3a admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:24.600356 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50fbb6a7-bf38-4c1e-ba6d-43923782df3a admin admin] PATH_INFO: `/v3/projects/e1ccdbaa29bd464ba64ad77ff7a00244` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:24.608452 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50fbb6a7-bf38-4c1e-ba6d-43923782df3a admin admin] RBAC: Authorizing `identity:delete_project(project_id=e1ccdbaa29bd464ba64ad77ff7a00244)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:24.610518 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50fbb6a7-bf38-4c1e-ba6d-43923782df3a admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:24.671738 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-50fbb6a7-bf38-4c1e-ba6d-43923782df3a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e1ccdbaa29bd464ba64ad77ff7a00244', 'request_id': 'req-50fbb6a7-bf38-4c1e-ba6d-43923782df3a'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:24.680699 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-50fbb6a7-bf38-4c1e-ba6d-43923782df3a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e1ccdbaa29bd464ba64ad77ff7a00244', 'request_id': 'req-50fbb6a7-bf38-4c1e-ba6d-43923782df3a'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:24.687370 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-50fbb6a7-bf38-4c1e-ba6d-43923782df3a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e1ccdbaa29bd464ba64ad77ff7a00244', 'request_id': 'req-50fbb6a7-bf38-4c1e-ba6d-43923782df3a'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:24.706154 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-50fbb6a7-bf38-4c1e-ba6d-43923782df3a admin admin] The token cache is being invalidate because project e1ccdbaa29bd464ba64ad77ff7a00244 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81453) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 00:58:24.706991 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-50fbb6a7-bf38-4c1e-ba6d-43923782df3a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:24.712057 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-50fbb6a7-bf38-4c1e-ba6d-43923782df3a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:24.713940 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1155/2344] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 00:58:24 2022] DELETE /identity/v3/projects/e1ccdbaa29bd464ba64ad77ff7a00244 => generated 0 bytes in 195 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:58:24.718534 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df4ab3ea-5c4c-49e3-8a09-542f382eb293 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:24.718990 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df4ab3ea-5c4c-49e3-8a09-542f382eb293 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:24.719130 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df4ab3ea-5c4c-49e3-8a09-542f382eb293 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:24.740731 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-df4ab3ea-5c4c-49e3-8a09-542f382eb293 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:58:24.742004 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-df4ab3ea-5c4c-49e3-8a09-542f382eb293 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:58:24.825378 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1190/2345] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:58:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:58:24.835007 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03a7825-0527-4b83-af37-77312c4b7f01 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:24.835391 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03a7825-0527-4b83-af37-77312c4b7f01 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:24.835391 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03a7825-0527-4b83-af37-77312c4b7f01 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:24.860641 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-d03a7825-0527-4b83-af37-77312c4b7f01 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:58:24.861885 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-d03a7825-0527-4b83-af37-77312c4b7f01 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:58:24.872769 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1156/2346] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:58:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:58:24.877497 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df4ab3ea-5c4c-49e3-8a09-542f382eb293 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:24.883926 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:58:24.891841 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df4ab3ea-5c4c-49e3-8a09-542f382eb293 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:24.903286 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bc51941-ffb4-449b-82db-8285dd023a02 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:24.904027 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc51941-ffb4-449b-82db-8285dd023a02 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:24.904368 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc51941-ffb4-449b-82db-8285dd023a02 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:24.904705 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc51941-ffb4-449b-82db-8285dd023a02 admin admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:24.905438 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bc51941-ffb4-449b-82db-8285dd023a02 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:58:24.905801 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bc51941-ffb4-449b-82db-8285dd023a02 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:24.907891 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bc51941-ffb4-449b-82db-8285dd023a02 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:24.935290 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1191/2347] 10.222.0.33 () {58 vars in 1153 bytes} [Sun Oct 9 00:58:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:58:24.936502 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d03a7825-0527-4b83-af37-77312c4b7f01 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:24.946798 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d03a7825-0527-4b83-af37-77312c4b7f01 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:24.952494 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-187f9acb-5a02-461e-9a68-d9b22e859ee5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:24.953177 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-187f9acb-5a02-461e-9a68-d9b22e859ee5 admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:24.953526 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-187f9acb-5a02-461e-9a68-d9b22e859ee5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:24.953886 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-187f9acb-5a02-461e-9a68-d9b22e859ee5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:24.954578 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-187f9acb-5a02-461e-9a68-d9b22e859ee5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:24.957283 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-187f9acb-5a02-461e-9a68-d9b22e859ee5 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:24.990459 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1157/2348] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 00:58:24 2022] POST /identity/v3/projects => generated 375 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:58:24.992051 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bc51941-ffb4-449b-82db-8285dd023a02 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:25.005376 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bc51941-ffb4-449b-82db-8285dd023a02 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:25.069896 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a266d737-1e0d-4c4c-b763-d5921b862996 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:25.070501 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a266d737-1e0d-4c4c-b763-d5921b862996 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:25.070781 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a266d737-1e0d-4c4c-b763-d5921b862996 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:25.071053 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a266d737-1e0d-4c4c-b763-d5921b862996 admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:25.072000 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a266d737-1e0d-4c4c-b763-d5921b862996 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:25.077441 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a266d737-1e0d-4c4c-b763-d5921b862996 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:25.105149 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1192/2349] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:58:24 2022] POST /identity/v3/users => generated 288 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:58:25.109083 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-187f9acb-5a02-461e-9a68-d9b22e859ee5 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:25.115351 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:58:25.121526 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-187f9acb-5a02-461e-9a68-d9b22e859ee5 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:25.129463 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1023cfb8-0051-430f-8044-113b75289957 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:25.130372 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1023cfb8-0051-430f-8044-113b75289957 admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:25.130780 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1023cfb8-0051-430f-8044-113b75289957 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:25.131049 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1023cfb8-0051-430f-8044-113b75289957 admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:25.131779 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1023cfb8-0051-430f-8044-113b75289957 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:25.134785 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1023cfb8-0051-430f-8044-113b75289957 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:25.151650 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1158/2350] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:58:25 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:58:25.155202 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a266d737-1e0d-4c4c-b763-d5921b862996 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:25.169414 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a266d737-1e0d-4c4c-b763-d5921b862996 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:25.178363 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e166e9a-9d27-4812-97ea-db0f8b0085e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:25.180128 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e166e9a-9d27-4812-97ea-db0f8b0085e7 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:25.180616 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e166e9a-9d27-4812-97ea-db0f8b0085e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:25.181238 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e166e9a-9d27-4812-97ea-db0f8b0085e7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:25.182112 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e166e9a-9d27-4812-97ea-db0f8b0085e7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:25.184113 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e166e9a-9d27-4812-97ea-db0f8b0085e7 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:25.203987 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1193/2351] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:58:25 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:58:25.215247 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1023cfb8-0051-430f-8044-113b75289957 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:25.233768 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1023cfb8-0051-430f-8044-113b75289957 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:25.241356 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd98537c-68a2-4b30-89c5-ed7726384206 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:25.242535 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd98537c-68a2-4b30-89c5-ed7726384206 admin admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:25.242921 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd98537c-68a2-4b30-89c5-ed7726384206 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:25.243361 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd98537c-68a2-4b30-89c5-ed7726384206 admin admin] PATH_INFO: `/v3/projects/89d50d3addae4862bae5a8c8117896c0/users/76feab2c01004e30924beaa195c6cbc1/roles/588016933bab436d8b7c3ac76827356d` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:25.259403 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd98537c-68a2-4b30-89c5-ed7726384206 admin admin] RBAC: Authorizing `identity:create_grant(project_id=89d50d3addae4862bae5a8c8117896c0, user_id=76feab2c01004e30924beaa195c6cbc1, role_id=588016933bab436d8b7c3ac76827356d)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:25.262049 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd98537c-68a2-4b30-89c5-ed7726384206 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:25.274574 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1159/2352] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:58:25 2022] PUT /identity/v3/projects/89d50d3addae4862bae5a8c8117896c0/users/76feab2c01004e30924beaa195c6cbc1/roles/588016933bab436d8b7c3ac76827356d => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:58:25.285221 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf5fd2bc-dbfc-4dc7-8c4d-362c57d35d96 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:25.285529 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf5fd2bc-dbfc-4dc7-8c4d-362c57d35d96 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:25.285904 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf5fd2bc-dbfc-4dc7-8c4d-362c57d35d96 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:25.315180 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-bf5fd2bc-dbfc-4dc7-8c4d-362c57d35d96 None None] MFA Rules not processed for user `76feab2c01004e30924beaa195c6cbc1`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:58:25.317232 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-bf5fd2bc-dbfc-4dc7-8c4d-362c57d35d96 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:58:25.399061 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1194/2353] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:58:25 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 09 00:58:25.403503 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd98537c-68a2-4b30-89c5-ed7726384206 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:25.464007 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd98537c-68a2-4b30-89c5-ed7726384206 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:25.468678 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ef30a66-09ba-4b9b-b31b-ab15b954fb27 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:25.469272 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ef30a66-09ba-4b9b-b31b-ab15b954fb27 admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:25.469556 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ef30a66-09ba-4b9b-b31b-ab15b954fb27 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:25.469830 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ef30a66-09ba-4b9b-b31b-ab15b954fb27 admin admin] PATH_INFO: `/v3/projects` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:25.470445 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ef30a66-09ba-4b9b-b31b-ab15b954fb27 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:25.472906 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ef30a66-09ba-4b9b-b31b-ab15b954fb27 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:25.502259 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1160/2354] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 00:58:25 2022] POST /identity/v3/projects => generated 375 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:58:25.506513 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf5fd2bc-dbfc-4dc7-8c4d-362c57d35d96 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:25.519262 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf5fd2bc-dbfc-4dc7-8c4d-362c57d35d96 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:25.580742 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:58:25.588452 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79d469c7-b73a-42b7-80cf-9ec28ef01daa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:25.589545 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d469c7-b73a-42b7-80cf-9ec28ef01daa admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:25.589883 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d469c7-b73a-42b7-80cf-9ec28ef01daa admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:25.590842 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d469c7-b73a-42b7-80cf-9ec28ef01daa admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:25.591963 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79d469c7-b73a-42b7-80cf-9ec28ef01daa admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:25.594548 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79d469c7-b73a-42b7-80cf-9ec28ef01daa admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:25.617446 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1195/2355] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:58:25 2022] POST /identity/v3/users => generated 294 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:58:25.621595 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ef30a66-09ba-4b9b-b31b-ab15b954fb27 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:25.632013 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ef30a66-09ba-4b9b-b31b-ab15b954fb27 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:25.636716 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0cd1a85-1f1a-4bdb-b0af-9ecd6e56f067 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:25.637415 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0cd1a85-1f1a-4bdb-b0af-9ecd6e56f067 admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:25.637767 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0cd1a85-1f1a-4bdb-b0af-9ecd6e56f067 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:25.638556 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0cd1a85-1f1a-4bdb-b0af-9ecd6e56f067 admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:25.638854 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0cd1a85-1f1a-4bdb-b0af-9ecd6e56f067 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:25.640626 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0cd1a85-1f1a-4bdb-b0af-9ecd6e56f067 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:25.651440 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1161/2356] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:58:25 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:58:25.655707 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79d469c7-b73a-42b7-80cf-9ec28ef01daa admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:25.666632 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79d469c7-b73a-42b7-80cf-9ec28ef01daa admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:25.671397 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4598eea3-16d3-4536-8fbc-cfb3a8fb2b58 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:25.672073 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4598eea3-16d3-4536-8fbc-cfb3a8fb2b58 admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:25.672351 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4598eea3-16d3-4536-8fbc-cfb3a8fb2b58 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:25.672622 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4598eea3-16d3-4536-8fbc-cfb3a8fb2b58 admin admin] PATH_INFO: `/v3/projects/dff161b33ef74fb0bf4d7f5bca33cc8b/users/0d007a3e5d54411f928806766f811718/roles/092036ee46ac427baf4d56468575f697` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:25.693722 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4598eea3-16d3-4536-8fbc-cfb3a8fb2b58 admin admin] RBAC: Authorizing `identity:create_grant(project_id=dff161b33ef74fb0bf4d7f5bca33cc8b, user_id=0d007a3e5d54411f928806766f811718, role_id=092036ee46ac427baf4d56468575f697)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:25.696388 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4598eea3-16d3-4536-8fbc-cfb3a8fb2b58 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:25.724720 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1196/2357] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:58:25 2022] PUT /identity/v3/projects/dff161b33ef74fb0bf4d7f5bca33cc8b/users/0d007a3e5d54411f928806766f811718/roles/092036ee46ac427baf4d56468575f697 => generated 0 bytes in 69 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:58:25.728653 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0cd1a85-1f1a-4bdb-b0af-9ecd6e56f067 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:25.740135 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0cd1a85-1f1a-4bdb-b0af-9ecd6e56f067 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:25.799697 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9999892-da00-4e52-91d3-7af540e99bdf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:25.800825 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9999892-da00-4e52-91d3-7af540e99bdf admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:25.801106 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9999892-da00-4e52-91d3-7af540e99bdf admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:25.801380 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9999892-da00-4e52-91d3-7af540e99bdf admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:25.802014 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9999892-da00-4e52-91d3-7af540e99bdf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:25.803638 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9999892-da00-4e52-91d3-7af540e99bdf admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:25.815101 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1162/2358] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:58:25 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:58:25.819481 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4598eea3-16d3-4536-8fbc-cfb3a8fb2b58 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:25.830573 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4598eea3-16d3-4536-8fbc-cfb3a8fb2b58 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:25.835464 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3a92986-a20b-4c6e-a348-7c65399c040c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:25.836180 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a92986-a20b-4c6e-a348-7c65399c040c admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:25.836463 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a92986-a20b-4c6e-a348-7c65399c040c admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:25.836736 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a92986-a20b-4c6e-a348-7c65399c040c admin admin] PATH_INFO: `/v3/projects/dff161b33ef74fb0bf4d7f5bca33cc8b/users/0d007a3e5d54411f928806766f811718/roles/092036ee46ac427baf4d56468575f697` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:25.838699 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3a92986-a20b-4c6e-a348-7c65399c040c admin admin] RBAC: Authorizing `identity:create_grant(project_id=dff161b33ef74fb0bf4d7f5bca33cc8b, user_id=0d007a3e5d54411f928806766f811718, role_id=092036ee46ac427baf4d56468575f697)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:25.841479 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3a92986-a20b-4c6e-a348-7c65399c040c admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:25.854788 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1197/2359] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:58:25 2022] PUT /identity/v3/projects/dff161b33ef74fb0bf4d7f5bca33cc8b/users/0d007a3e5d54411f928806766f811718/roles/092036ee46ac427baf4d56468575f697 => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 09 00:58:25.869474 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f9c07a-3a4e-420b-af04-fdf446c0708c None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:25.870595 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f9c07a-3a4e-420b-af04-fdf446c0708c None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:25.871117 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f9c07a-3a4e-420b-af04-fdf446c0708c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:25.905674 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-84f9c07a-3a4e-420b-af04-fdf446c0708c None None] MFA Rules not processed for user `0d007a3e5d54411f928806766f811718`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:58:25.908618 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-84f9c07a-3a4e-420b-af04-fdf446c0708c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:58:25.970338 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:58:26.043241 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1163/2360] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:58:25 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:58:26.054843 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3a92986-a20b-4c6e-a348-7c65399c040c admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:26.056586 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-e3a92986-a20b-4c6e-a348-7c65399c040c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:58:26.124179 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3a92986-a20b-4c6e-a348-7c65399c040c admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:26.127840 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41f374a5-9751-4556-b460-ee82c23fb164 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3f48335668a4e8f934ac11ff66cd308', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:26.128473 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41f374a5-9751-4556-b460-ee82c23fb164 service cinder] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:26.128812 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41f374a5-9751-4556-b460-ee82c23fb164 service cinder] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:26.129084 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41f374a5-9751-4556-b460-ee82c23fb164 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:26.186369 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41f374a5-9751-4556-b460-ee82c23fb164 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:26.187866 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41f374a5-9751-4556-b460-ee82c23fb164 service cinder] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:26.196696 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1198/2361] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:58:26 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 142 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:58:28.778431 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84f9c07a-3a4e-420b-af04-fdf446c0708c None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:28.868724 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84f9c07a-3a4e-420b-af04-fdf446c0708c None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:28.873684 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07d13be1-d40e-4530-a878-70795049c632 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:28.874327 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07d13be1-d40e-4530-a878-70795049c632 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:28.874612 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07d13be1-d40e-4530-a878-70795049c632 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:28.875095 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07d13be1-d40e-4530-a878-70795049c632 admin admin] PATH_INFO: `/v3/users/76feab2c01004e30924beaa195c6cbc1` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:28.876273 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07d13be1-d40e-4530-a878-70795049c632 admin admin] RBAC: Authorizing `identity:delete_user(user_id=76feab2c01004e30924beaa195c6cbc1)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:28.880133 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07d13be1-d40e-4530-a878-70795049c632 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:28.940516 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-07d13be1-d40e-4530-a878-70795049c632 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '76feab2c01004e30924beaa195c6cbc1'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:28.952338 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-07d13be1-d40e-4530-a878-70795049c632 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '76feab2c01004e30924beaa195c6cbc1'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:28.953501 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-07d13be1-d40e-4530-a878-70795049c632 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '76feab2c01004e30924beaa195c6cbc1'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:28.965425 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-07d13be1-d40e-4530-a878-70795049c632 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '76feab2c01004e30924beaa195c6cbc1'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:28.981502 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-07d13be1-d40e-4530-a878-70795049c632 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '76feab2c01004e30924beaa195c6cbc1'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:28.996263 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1164/2362] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 00:58:28 2022] DELETE /identity/v3/users/76feab2c01004e30924beaa195c6cbc1 => generated 0 bytes in 218 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:58:29.000720 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b68f38-6897-4f98-a982-8f1c661b6522 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:29.001245 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b68f38-6897-4f98-a982-8f1c661b6522 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:29.001441 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b68f38-6897-4f98-a982-8f1c661b6522 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:29.028075 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-c8b68f38-6897-4f98-a982-8f1c661b6522 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:58:29.029639 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-c8b68f38-6897-4f98-a982-8f1c661b6522 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:58:29.129277 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1199/2363] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:58:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:58:29.157397 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07d13be1-d40e-4530-a878-70795049c632 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:29.159273 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-07d13be1-d40e-4530-a878-70795049c632 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:58:29.253708 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:58:29.274705 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07d13be1-d40e-4530-a878-70795049c632 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:29.286733 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-264933c3-ec2c-48a3-b6df-912d902811ae service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:29.287398 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-264933c3-ec2c-48a3-b6df-912d902811ae service neutron] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:29.287701 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-264933c3-ec2c-48a3-b6df-912d902811ae service neutron] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:29.288637 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-264933c3-ec2c-48a3-b6df-912d902811ae service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:29.299924 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-264933c3-ec2c-48a3-b6df-912d902811ae service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:29.302402 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-264933c3-ec2c-48a3-b6df-912d902811ae service neutron] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:29.312569 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1165/2364] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:58:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 156 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:58:29.737650 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce4d8706-2d59-4ab4-8564-276663246911 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:29.737998 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce4d8706-2d59-4ab4-8564-276663246911 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:29.738200 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce4d8706-2d59-4ab4-8564-276663246911 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:29.747966 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:58:29.761957 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-ce4d8706-2d59-4ab4-8564-276663246911 None None] MFA Rules not processed for user `04bc22d558c2466ba43a456d08bdd6bd`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:58:29.763652 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-ce4d8706-2d59-4ab4-8564-276663246911 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:58:29.856987 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1200/2365] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:58:29 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:58:30.376801 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-264933c3-ec2c-48a3-b6df-912d902811ae service neutron] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:30.378901 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-264933c3-ec2c-48a3-b6df-912d902811ae service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:58:30.392323 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-264933c3-ec2c-48a3-b6df-912d902811ae service neutron] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:30.424329 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49643350-d90c-45bd-b9f3-1f37d50bd4fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:30.426434 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49643350-d90c-45bd-b9f3-1f37d50bd4fd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:30.426854 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49643350-d90c-45bd-b9f3-1f37d50bd4fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:30.427225 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49643350-d90c-45bd-b9f3-1f37d50bd4fd admin admin] PATH_INFO: `/v3/projects/89d50d3addae4862bae5a8c8117896c0` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:30.428524 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49643350-d90c-45bd-b9f3-1f37d50bd4fd admin admin] RBAC: Authorizing `identity:delete_project(project_id=89d50d3addae4862bae5a8c8117896c0)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:30.430912 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49643350-d90c-45bd-b9f3-1f37d50bd4fd admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:30.495135 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-49643350-d90c-45bd-b9f3-1f37d50bd4fd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '89d50d3addae4862bae5a8c8117896c0', 'request_id': 'req-49643350-d90c-45bd-b9f3-1f37d50bd4fd'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:30.504910 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-49643350-d90c-45bd-b9f3-1f37d50bd4fd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '89d50d3addae4862bae5a8c8117896c0', 'request_id': 'req-49643350-d90c-45bd-b9f3-1f37d50bd4fd'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:30.512016 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-49643350-d90c-45bd-b9f3-1f37d50bd4fd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '89d50d3addae4862bae5a8c8117896c0', 'request_id': 'req-49643350-d90c-45bd-b9f3-1f37d50bd4fd'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:30.529220 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-49643350-d90c-45bd-b9f3-1f37d50bd4fd admin admin] The token cache is being invalidate because project 89d50d3addae4862bae5a8c8117896c0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81453) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 00:58:30.529640 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-49643350-d90c-45bd-b9f3-1f37d50bd4fd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:30.531028 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-49643350-d90c-45bd-b9f3-1f37d50bd4fd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:30.533002 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1166/2366] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 00:58:30 2022] DELETE /identity/v3/projects/89d50d3addae4862bae5a8c8117896c0 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:58:30.537763 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce4d8706-2d59-4ab4-8564-276663246911 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:30.553473 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce4d8706-2d59-4ab4-8564-276663246911 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:30.652739 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf7a69e0-df45-4cd5-96f4-22b65665f171 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:30.653515 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf7a69e0-df45-4cd5-96f4-22b65665f171 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:30.654003 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf7a69e0-df45-4cd5-96f4-22b65665f171 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:30.654431 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf7a69e0-df45-4cd5-96f4-22b65665f171 admin admin] PATH_INFO: `/v3/users/0d007a3e5d54411f928806766f811718` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:30.656063 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf7a69e0-df45-4cd5-96f4-22b65665f171 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0d007a3e5d54411f928806766f811718)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:30.658161 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf7a69e0-df45-4cd5-96f4-22b65665f171 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:30.732298 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-bf7a69e0-df45-4cd5-96f4-22b65665f171 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0d007a3e5d54411f928806766f811718'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:30.740803 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-bf7a69e0-df45-4cd5-96f4-22b65665f171 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0d007a3e5d54411f928806766f811718'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:30.741746 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-bf7a69e0-df45-4cd5-96f4-22b65665f171 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0d007a3e5d54411f928806766f811718'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:30.742567 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-bf7a69e0-df45-4cd5-96f4-22b65665f171 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0d007a3e5d54411f928806766f811718'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:30.763093 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-bf7a69e0-df45-4cd5-96f4-22b65665f171 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0d007a3e5d54411f928806766f811718'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:30.793035 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1201/2367] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 00:58:30 2022] DELETE /identity/v3/users/0d007a3e5d54411f928806766f811718 => generated 0 bytes in 256 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:58:31.062443 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b60e991-e0ab-44ca-b17c-e2a69c17b5ef None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:31.063169 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b60e991-e0ab-44ca-b17c-e2a69c17b5ef None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:31.063694 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b60e991-e0ab-44ca-b17c-e2a69c17b5ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:31.089078 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-2b60e991-e0ab-44ca-b17c-e2a69c17b5ef None None] MFA Rules not processed for user `438446d1e6b3402bb18b547c089391f8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:58:31.091681 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-2b60e991-e0ab-44ca-b17c-e2a69c17b5ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:58:31.164179 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf7a69e0-df45-4cd5-96f4-22b65665f171 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:31.166459 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-bf7a69e0-df45-4cd5-96f4-22b65665f171 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:58:31.234115 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1167/2368] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:58:31 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:58:31.261270 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:58:31.329526 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf7a69e0-df45-4cd5-96f4-22b65665f171 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:31.338165 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afb76a35-98ab-4430-baf8-e952f9fd2056 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:31.339143 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afb76a35-98ab-4430-baf8-e952f9fd2056 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:31.339525 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afb76a35-98ab-4430-baf8-e952f9fd2056 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:31.340598 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afb76a35-98ab-4430-baf8-e952f9fd2056 admin admin] PATH_INFO: `/v3/projects/dff161b33ef74fb0bf4d7f5bca33cc8b` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:31.341872 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afb76a35-98ab-4430-baf8-e952f9fd2056 admin admin] RBAC: Authorizing `identity:delete_project(project_id=dff161b33ef74fb0bf4d7f5bca33cc8b)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:31.344196 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afb76a35-98ab-4430-baf8-e952f9fd2056 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:31.434134 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-afb76a35-98ab-4430-baf8-e952f9fd2056 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'dff161b33ef74fb0bf4d7f5bca33cc8b', 'request_id': 'req-afb76a35-98ab-4430-baf8-e952f9fd2056'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:31.444960 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-afb76a35-98ab-4430-baf8-e952f9fd2056 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'dff161b33ef74fb0bf4d7f5bca33cc8b', 'request_id': 'req-afb76a35-98ab-4430-baf8-e952f9fd2056'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:31.461980 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-afb76a35-98ab-4430-baf8-e952f9fd2056 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'dff161b33ef74fb0bf4d7f5bca33cc8b', 'request_id': 'req-afb76a35-98ab-4430-baf8-e952f9fd2056'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:31.486266 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-afb76a35-98ab-4430-baf8-e952f9fd2056 admin admin] The token cache is being invalidate because project dff161b33ef74fb0bf4d7f5bca33cc8b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 00:58:31.486548 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-afb76a35-98ab-4430-baf8-e952f9fd2056 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:31.487809 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-afb76a35-98ab-4430-baf8-e952f9fd2056 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:31.489668 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1202/2369] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 00:58:31 2022] DELETE /identity/v3/projects/dff161b33ef74fb0bf4d7f5bca33cc8b => generated 0 bytes in 327 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:58:31.494875 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d8e72e-9417-429a-9f9e-4016e22be936 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:31.495423 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d8e72e-9417-429a-9f9e-4016e22be936 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:31.495423 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d8e72e-9417-429a-9f9e-4016e22be936 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:31.519004 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-d4d8e72e-9417-429a-9f9e-4016e22be936 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:58:31.520619 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-d4d8e72e-9417-429a-9f9e-4016e22be936 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:58:31.670644 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1168/2370] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:58:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:58:31.679374 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3351c2cf-39a5-443d-8b95-5cd6f26fb11a None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:31.680206 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3351c2cf-39a5-443d-8b95-5cd6f26fb11a None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:31.680594 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3351c2cf-39a5-443d-8b95-5cd6f26fb11a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:31.723173 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-3351c2cf-39a5-443d-8b95-5cd6f26fb11a None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:58:31.723173 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-3351c2cf-39a5-443d-8b95-5cd6f26fb11a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:58:31.735306 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1203/2371] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:58:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 57 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:58:31.740442 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4d8e72e-9417-429a-9f9e-4016e22be936 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:31.752994 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4d8e72e-9417-429a-9f9e-4016e22be936 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:31.763740 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70f8923b-2fe1-4392-845e-f7d189c8ab78 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:31.764769 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f8923b-2fe1-4392-845e-f7d189c8ab78 admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:31.765208 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f8923b-2fe1-4392-845e-f7d189c8ab78 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:31.765637 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f8923b-2fe1-4392-845e-f7d189c8ab78 admin admin] PATH_INFO: `/v3/domains` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:31.766746 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70f8923b-2fe1-4392-845e-f7d189c8ab78 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:58:31.767238 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70f8923b-2fe1-4392-845e-f7d189c8ab78 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:31.770146 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70f8923b-2fe1-4392-845e-f7d189c8ab78 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:31.781382 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:58:31.793005 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1169/2372] 10.222.0.33 () {58 vars in 1153 bytes} [Sun Oct 9 00:58:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:58:31.793946 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3351c2cf-39a5-443d-8b95-5cd6f26fb11a None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:31.804977 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3351c2cf-39a5-443d-8b95-5cd6f26fb11a None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:31.811545 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e114cac-2979-4171-884b-7b42ce54118d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:31.812401 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e114cac-2979-4171-884b-7b42ce54118d admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:31.812735 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e114cac-2979-4171-884b-7b42ce54118d admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:31.813010 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e114cac-2979-4171-884b-7b42ce54118d admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:31.813838 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e114cac-2979-4171-884b-7b42ce54118d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:31.820462 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e114cac-2979-4171-884b-7b42ce54118d admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:31.864889 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1204/2373] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 00:58:31 2022] POST /identity/v3/projects => generated 385 bytes in 71 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 09 00:58:31.865892 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70f8923b-2fe1-4392-845e-f7d189c8ab78 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:31.879213 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70f8923b-2fe1-4392-845e-f7d189c8ab78 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:31.950792 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ab55e3-220b-462b-9a66-94cfd5405558 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:31.951757 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ab55e3-220b-462b-9a66-94cfd5405558 admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:31.952049 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ab55e3-220b-462b-9a66-94cfd5405558 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:31.952315 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ab55e3-220b-462b-9a66-94cfd5405558 admin admin] PATH_INFO: `/v3/users` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:31.952951 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71ab55e3-220b-462b-9a66-94cfd5405558 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:31.954866 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71ab55e3-220b-462b-9a66-94cfd5405558 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:32.004024 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1170/2374] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:58:31 2022] POST /identity/v3/users => generated 293 bytes in 140 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 09 00:58:32.004669 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e114cac-2979-4171-884b-7b42ce54118d admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:32.019167 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e114cac-2979-4171-884b-7b42ce54118d admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:32.024431 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f32a8eab-a30c-44ab-ba4a-5e0cecd733b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:32.025053 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f32a8eab-a30c-44ab-ba4a-5e0cecd733b4 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:32.025362 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f32a8eab-a30c-44ab-ba4a-5e0cecd733b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:32.025611 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f32a8eab-a30c-44ab-ba4a-5e0cecd733b4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:32.026335 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f32a8eab-a30c-44ab-ba4a-5e0cecd733b4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:32.028080 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f32a8eab-a30c-44ab-ba4a-5e0cecd733b4 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:32.041181 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1205/2375] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:58:32 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 09 00:58:32.056259 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ab55e3-220b-462b-9a66-94cfd5405558 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:32.096289 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ab55e3-220b-462b-9a66-94cfd5405558 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:32.105742 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-138a8b4a-3698-40ad-b310-48eb96c63d53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:32.106503 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-138a8b4a-3698-40ad-b310-48eb96c63d53 admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:32.106810 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-138a8b4a-3698-40ad-b310-48eb96c63d53 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:32.107066 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-138a8b4a-3698-40ad-b310-48eb96c63d53 admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:32.107769 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-138a8b4a-3698-40ad-b310-48eb96c63d53 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:32.110896 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-138a8b4a-3698-40ad-b310-48eb96c63d53 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:32.124327 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1171/2376] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:58:32 2022] GET /identity/v3/roles => generated 1420 bytes in 68 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:58:32.128534 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f32a8eab-a30c-44ab-ba4a-5e0cecd733b4 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:32.139995 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f32a8eab-a30c-44ab-ba4a-5e0cecd733b4 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:32.147860 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f3efb2e-685f-4869-8f81-5ee7f6e0a28c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:32.148537 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f3efb2e-685f-4869-8f81-5ee7f6e0a28c admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:32.148844 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f3efb2e-685f-4869-8f81-5ee7f6e0a28c admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:32.149124 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f3efb2e-685f-4869-8f81-5ee7f6e0a28c admin admin] PATH_INFO: `/v3/projects/41cc12d5923847858268ee2c8be05164/users/1de4706a5d0a4f6283fb99a75552e3c2/roles/588016933bab436d8b7c3ac76827356d` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:32.174747 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f3efb2e-685f-4869-8f81-5ee7f6e0a28c admin admin] RBAC: Authorizing `identity:create_grant(project_id=41cc12d5923847858268ee2c8be05164, user_id=1de4706a5d0a4f6283fb99a75552e3c2, role_id=588016933bab436d8b7c3ac76827356d)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:32.177590 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f3efb2e-685f-4869-8f81-5ee7f6e0a28c admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:32.195882 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1206/2377] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:58:32 2022] PUT /identity/v3/projects/41cc12d5923847858268ee2c8be05164/users/1de4706a5d0a4f6283fb99a75552e3c2/roles/588016933bab436d8b7c3ac76827356d => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 09 00:58:32.204675 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf16b60-ef87-484c-b5bb-ad6bf68024aa None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:32.205273 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf16b60-ef87-484c-b5bb-ad6bf68024aa None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:32.205784 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf16b60-ef87-484c-b5bb-ad6bf68024aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:32.243727 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-daf16b60-ef87-484c-b5bb-ad6bf68024aa None None] MFA Rules not processed for user `1de4706a5d0a4f6283fb99a75552e3c2`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:58:32.245102 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-daf16b60-ef87-484c-b5bb-ad6bf68024aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:58:32.348975 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1172/2378] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:58:32 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:58:32.353516 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f3efb2e-685f-4869-8f81-5ee7f6e0a28c admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:32.431911 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:58:32.441385 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f3efb2e-685f-4869-8f81-5ee7f6e0a28c admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:32.462882 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f2b00ce-b2f6-472b-a154-e307131fe795 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:32.463910 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f2b00ce-b2f6-472b-a154-e307131fe795 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:32.464406 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f2b00ce-b2f6-472b-a154-e307131fe795 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:32.464861 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f2b00ce-b2f6-472b-a154-e307131fe795 admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:32.465721 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f2b00ce-b2f6-472b-a154-e307131fe795 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:32.468090 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f2b00ce-b2f6-472b-a154-e307131fe795 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:32.514019 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1207/2379] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 00:58:32 2022] POST /identity/v3/projects => generated 383 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:58:32.519257 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daf16b60-ef87-484c-b5bb-ad6bf68024aa None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:32.535675 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daf16b60-ef87-484c-b5bb-ad6bf68024aa None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:32.606239 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-717a8ca9-73b9-42ff-9334-a19bdaf3595f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:32.606857 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-717a8ca9-73b9-42ff-9334-a19bdaf3595f admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:32.607349 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-717a8ca9-73b9-42ff-9334-a19bdaf3595f admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:32.607462 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-717a8ca9-73b9-42ff-9334-a19bdaf3595f admin admin] PATH_INFO: `/v3/users` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:32.608194 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-717a8ca9-73b9-42ff-9334-a19bdaf3595f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:32.609984 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-717a8ca9-73b9-42ff-9334-a19bdaf3595f admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:32.630924 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:58:32.644481 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f2b00ce-b2f6-472b-a154-e307131fe795 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:32.644741 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1173/2380] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:58:32 2022] POST /identity/v3/users => generated 292 bytes in 127 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:58:32.657308 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f2b00ce-b2f6-472b-a154-e307131fe795 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:32.667879 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ccbc93d-f349-4c15-bfeb-505bc3497f11 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:32.668671 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ccbc93d-f349-4c15-bfeb-505bc3497f11 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:32.669075 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ccbc93d-f349-4c15-bfeb-505bc3497f11 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:32.669386 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ccbc93d-f349-4c15-bfeb-505bc3497f11 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:32.670550 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ccbc93d-f349-4c15-bfeb-505bc3497f11 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:32.672615 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ccbc93d-f349-4c15-bfeb-505bc3497f11 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:32.688892 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1208/2381] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:58:32 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 09 00:58:32.693246 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-717a8ca9-73b9-42ff-9334-a19bdaf3595f admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:32.731011 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-717a8ca9-73b9-42ff-9334-a19bdaf3595f admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:32.746098 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57cf6852-1d6c-4bcf-a356-6d330dd42545 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:32.747120 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57cf6852-1d6c-4bcf-a356-6d330dd42545 admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:32.747313 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57cf6852-1d6c-4bcf-a356-6d330dd42545 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:32.749541 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57cf6852-1d6c-4bcf-a356-6d330dd42545 admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:32.750443 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57cf6852-1d6c-4bcf-a356-6d330dd42545 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:32.756972 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57cf6852-1d6c-4bcf-a356-6d330dd42545 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:32.802801 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ccbc93d-f349-4c15-bfeb-505bc3497f11 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:32.809233 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1174/2382] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:58:32 2022] GET /identity/v3/roles => generated 1420 bytes in 108 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:58:32.816739 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ccbc93d-f349-4c15-bfeb-505bc3497f11 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:32.828360 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81f4fb6c-07b9-482b-8884-8f991354a76f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:32.831621 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f4fb6c-07b9-482b-8884-8f991354a76f admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:32.832079 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f4fb6c-07b9-482b-8884-8f991354a76f admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:32.834292 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f4fb6c-07b9-482b-8884-8f991354a76f admin admin] PATH_INFO: `/v3/projects/15814fc60c8c40188bbce5f0ce50db10/users/d8899e1ca4a941a2ac01d542801fc30d/roles/588016933bab436d8b7c3ac76827356d` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:32.870877 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81f4fb6c-07b9-482b-8884-8f991354a76f admin admin] RBAC: Authorizing `identity:create_grant(project_id=15814fc60c8c40188bbce5f0ce50db10, user_id=d8899e1ca4a941a2ac01d542801fc30d, role_id=588016933bab436d8b7c3ac76827356d)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:32.876547 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81f4fb6c-07b9-482b-8884-8f991354a76f admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:32.921953 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1209/2383] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:58:32 2022] PUT /identity/v3/projects/15814fc60c8c40188bbce5f0ce50db10/users/d8899e1ca4a941a2ac01d542801fc30d/roles/588016933bab436d8b7c3ac76827356d => generated 0 bytes in 121 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:58:32.935538 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c608a97-6fa7-4ea1-b3c0-094b53a1766a None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:32.935890 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c608a97-6fa7-4ea1-b3c0-094b53a1766a None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:32.936175 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c608a97-6fa7-4ea1-b3c0-094b53a1766a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:32.986712 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-1c608a97-6fa7-4ea1-b3c0-094b53a1766a None None] MFA Rules not processed for user `d8899e1ca4a941a2ac01d542801fc30d`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:58:32.988159 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-1c608a97-6fa7-4ea1-b3c0-094b53a1766a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:58:33.138245 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1175/2384] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:58:32 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 205 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:58:33.142894 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81f4fb6c-07b9-482b-8884-8f991354a76f admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:33.242842 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81f4fb6c-07b9-482b-8884-8f991354a76f admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:33.247675 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11ab73ce-8d8d-47f6-901e-1255054b7f8d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:33.248210 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ab73ce-8d8d-47f6-901e-1255054b7f8d admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:33.248489 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ab73ce-8d8d-47f6-901e-1255054b7f8d admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:33.248766 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ab73ce-8d8d-47f6-901e-1255054b7f8d admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:33.249394 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11ab73ce-8d8d-47f6-901e-1255054b7f8d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:33.251332 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11ab73ce-8d8d-47f6-901e-1255054b7f8d admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:33.310174 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1210/2385] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 00:58:33 2022] POST /identity/v3/projects => generated 385 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:58:33.314735 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c608a97-6fa7-4ea1-b3c0-094b53a1766a None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:33.331838 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c608a97-6fa7-4ea1-b3c0-094b53a1766a None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:33.430946 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2b7a7e1-28d4-4b68-a06f-644ad97b8b5d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:33.431700 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2b7a7e1-28d4-4b68-a06f-644ad97b8b5d admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:33.432083 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2b7a7e1-28d4-4b68-a06f-644ad97b8b5d admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:33.432425 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2b7a7e1-28d4-4b68-a06f-644ad97b8b5d admin admin] PATH_INFO: `/v3/users` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:33.432995 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2b7a7e1-28d4-4b68-a06f-644ad97b8b5d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:33.435188 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2b7a7e1-28d4-4b68-a06f-644ad97b8b5d admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:33.460408 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1176/2386] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:58:33 2022] POST /identity/v3/users => generated 299 bytes in 146 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:58:33.464256 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11ab73ce-8d8d-47f6-901e-1255054b7f8d admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:33.474701 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11ab73ce-8d8d-47f6-901e-1255054b7f8d admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:33.480050 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c228c05e-339e-4dc6-a0a0-058d7e9847d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:33.480656 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c228c05e-339e-4dc6-a0a0-058d7e9847d3 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:33.480938 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c228c05e-339e-4dc6-a0a0-058d7e9847d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:33.481215 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c228c05e-339e-4dc6-a0a0-058d7e9847d3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:33.481863 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c228c05e-339e-4dc6-a0a0-058d7e9847d3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:33.483484 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c228c05e-339e-4dc6-a0a0-058d7e9847d3 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:33.495328 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1211/2387] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:58:33 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 09 00:58:33.499862 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2b7a7e1-28d4-4b68-a06f-644ad97b8b5d admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:33.510683 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2b7a7e1-28d4-4b68-a06f-644ad97b8b5d admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:33.515527 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ee45cc7-f0d7-4501-8e64-57f11d917072 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:33.516274 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee45cc7-f0d7-4501-8e64-57f11d917072 admin admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:33.516590 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee45cc7-f0d7-4501-8e64-57f11d917072 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:33.516869 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee45cc7-f0d7-4501-8e64-57f11d917072 admin admin] PATH_INFO: `/v3/projects/508281ed1de64e0ebaecd337b7d039f2/users/e287fcf5c6b5434fa9f9e86c51fce919/roles/092036ee46ac427baf4d56468575f697` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:33.533535 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ee45cc7-f0d7-4501-8e64-57f11d917072 admin admin] RBAC: Authorizing `identity:create_grant(project_id=508281ed1de64e0ebaecd337b7d039f2, user_id=e287fcf5c6b5434fa9f9e86c51fce919, role_id=092036ee46ac427baf4d56468575f697)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:33.536805 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ee45cc7-f0d7-4501-8e64-57f11d917072 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:33.550184 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1177/2388] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:58:33 2022] PUT /identity/v3/projects/508281ed1de64e0ebaecd337b7d039f2/users/e287fcf5c6b5434fa9f9e86c51fce919/roles/092036ee46ac427baf4d56468575f697 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:58:33.554088 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c228c05e-339e-4dc6-a0a0-058d7e9847d3 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:33.569661 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c228c05e-339e-4dc6-a0a0-058d7e9847d3 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:33.666687 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7214b66b-2776-4a2e-8efd-aee2ae29606f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:33.667313 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7214b66b-2776-4a2e-8efd-aee2ae29606f admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:33.667673 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7214b66b-2776-4a2e-8efd-aee2ae29606f admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:33.668033 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7214b66b-2776-4a2e-8efd-aee2ae29606f admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:33.668838 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7214b66b-2776-4a2e-8efd-aee2ae29606f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:33.671637 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7214b66b-2776-4a2e-8efd-aee2ae29606f admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:33.689254 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1212/2389] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:58:33 2022] GET /identity/v3/roles => generated 1420 bytes in 133 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:58:33.691504 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ee45cc7-f0d7-4501-8e64-57f11d917072 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:33.712632 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ee45cc7-f0d7-4501-8e64-57f11d917072 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:33.718487 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2939b2a-854c-4ce8-a628-7f2e47234089 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:33.719230 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2939b2a-854c-4ce8-a628-7f2e47234089 admin admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:33.719511 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2939b2a-854c-4ce8-a628-7f2e47234089 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:33.719848 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2939b2a-854c-4ce8-a628-7f2e47234089 admin admin] PATH_INFO: `/v3/projects/508281ed1de64e0ebaecd337b7d039f2/users/e287fcf5c6b5434fa9f9e86c51fce919/roles/092036ee46ac427baf4d56468575f697` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:33.724361 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2939b2a-854c-4ce8-a628-7f2e47234089 admin admin] RBAC: Authorizing `identity:create_grant(project_id=508281ed1de64e0ebaecd337b7d039f2, user_id=e287fcf5c6b5434fa9f9e86c51fce919, role_id=092036ee46ac427baf4d56468575f697)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:33.728274 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2939b2a-854c-4ce8-a628-7f2e47234089 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:33.743489 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1178/2390] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:58:33 2022] PUT /identity/v3/projects/508281ed1de64e0ebaecd337b7d039f2/users/e287fcf5c6b5434fa9f9e86c51fce919/roles/092036ee46ac427baf4d56468575f697 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:58:33.754207 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc329c6a-103f-40f3-982f-c1dc91e941cd None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:33.754502 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc329c6a-103f-40f3-982f-c1dc91e941cd None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:33.755172 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc329c6a-103f-40f3-982f-c1dc91e941cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:33.804645 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-cc329c6a-103f-40f3-982f-c1dc91e941cd None None] MFA Rules not processed for user `e287fcf5c6b5434fa9f9e86c51fce919`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:58:33.805907 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-cc329c6a-103f-40f3-982f-c1dc91e941cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:58:33.810417 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:58:33.912881 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1213/2391] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:58:33 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:58:33.926128 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2939b2a-854c-4ce8-a628-7f2e47234089 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:33.928279 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-a2939b2a-854c-4ce8-a628-7f2e47234089 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:58:34.001157 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:58:34.030273 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2939b2a-854c-4ce8-a628-7f2e47234089 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:34.037957 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83984486-c07f-4710-9eaa-e5085945db75 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3f48335668a4e8f934ac11ff66cd308', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:34.038791 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83984486-c07f-4710-9eaa-e5085945db75 service cinder] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:34.039443 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83984486-c07f-4710-9eaa-e5085945db75 service cinder] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:34.040014 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83984486-c07f-4710-9eaa-e5085945db75 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:34.110246 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83984486-c07f-4710-9eaa-e5085945db75 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:34.111714 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83984486-c07f-4710-9eaa-e5085945db75 service cinder] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:34.123183 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1179/2392] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:58:33 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 198 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:58:36.072147 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc329c6a-103f-40f3-982f-c1dc91e941cd None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:36.083180 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-cc329c6a-103f-40f3-982f-c1dc91e941cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:58:36.096973 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc329c6a-103f-40f3-982f-c1dc91e941cd None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:36.101376 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9be589a7-e489-45ff-b973-f4ec76e48c94 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3f48335668a4e8f934ac11ff66cd308', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:36.101985 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be589a7-e489-45ff-b973-f4ec76e48c94 service cinder] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:36.102676 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be589a7-e489-45ff-b973-f4ec76e48c94 service cinder] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:36.103138 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be589a7-e489-45ff-b973-f4ec76e48c94 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:36.204746 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9be589a7-e489-45ff-b973-f4ec76e48c94 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:36.206842 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9be589a7-e489-45ff-b973-f4ec76e48c94 service cinder] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:36.216751 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1214/2393] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:58:36 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 145 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:58:38.459515 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83984486-c07f-4710-9eaa-e5085945db75 service cinder] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:38.466030 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83984486-c07f-4710-9eaa-e5085945db75 service cinder] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:38.470606 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab0bd560-472c-4e56-baec-6e7a11339eb6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3f48335668a4e8f934ac11ff66cd308', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:38.471264 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab0bd560-472c-4e56-baec-6e7a11339eb6 service cinder] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:38.471749 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab0bd560-472c-4e56-baec-6e7a11339eb6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:38.472028 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab0bd560-472c-4e56-baec-6e7a11339eb6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:38.482435 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab0bd560-472c-4e56-baec-6e7a11339eb6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:38.483944 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab0bd560-472c-4e56-baec-6e7a11339eb6 service cinder] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:38.494371 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1180/2394] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:58:38 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:58:42.570668 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9be589a7-e489-45ff-b973-f4ec76e48c94 service cinder] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:42.644549 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9be589a7-e489-45ff-b973-f4ec76e48c94 service cinder] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:42.650546 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcf1eaa6-8647-47bf-9023-dbe7e4334576 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:42.650546 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcf1eaa6-8647-47bf-9023-dbe7e4334576 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:42.650546 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcf1eaa6-8647-47bf-9023-dbe7e4334576 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:42.650546 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcf1eaa6-8647-47bf-9023-dbe7e4334576 admin admin] PATH_INFO: `/v3/users/1de4706a5d0a4f6283fb99a75552e3c2` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:42.651417 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcf1eaa6-8647-47bf-9023-dbe7e4334576 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1de4706a5d0a4f6283fb99a75552e3c2)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:42.653339 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcf1eaa6-8647-47bf-9023-dbe7e4334576 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:42.696889 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=94,keystone:DELETE=6 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:58:42.703223 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-fcf1eaa6-8647-47bf-9023-dbe7e4334576 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1de4706a5d0a4f6283fb99a75552e3c2'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:42.711207 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-fcf1eaa6-8647-47bf-9023-dbe7e4334576 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1de4706a5d0a4f6283fb99a75552e3c2'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:42.712243 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-fcf1eaa6-8647-47bf-9023-dbe7e4334576 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1de4706a5d0a4f6283fb99a75552e3c2'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:42.713503 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-fcf1eaa6-8647-47bf-9023-dbe7e4334576 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1de4706a5d0a4f6283fb99a75552e3c2'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:42.721137 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-fcf1eaa6-8647-47bf-9023-dbe7e4334576 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1de4706a5d0a4f6283fb99a75552e3c2'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:42.740061 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1215/2395] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 00:58:42 2022] DELETE /identity/v3/users/1de4706a5d0a4f6283fb99a75552e3c2 => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:58:42.744680 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d53c1265-c6db-4f75-8c1a-0fb75e55d5f1 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:42.745246 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d53c1265-c6db-4f75-8c1a-0fb75e55d5f1 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:42.745246 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d53c1265-c6db-4f75-8c1a-0fb75e55d5f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:42.764466 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-d53c1265-c6db-4f75-8c1a-0fb75e55d5f1 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:58:42.765661 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-d53c1265-c6db-4f75-8c1a-0fb75e55d5f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:58:42.829191 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1181/2396] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:58:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:58:42.839773 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcf1eaa6-8647-47bf-9023-dbe7e4334576 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:42.841247 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-fcf1eaa6-8647-47bf-9023-dbe7e4334576 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:58:42.910787 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcf1eaa6-8647-47bf-9023-dbe7e4334576 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:42.919938 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca5cb127-91b3-4866-9aa0-32279501d542 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:42.920893 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca5cb127-91b3-4866-9aa0-32279501d542 service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:42.921173 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca5cb127-91b3-4866-9aa0-32279501d542 service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:42.921520 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca5cb127-91b3-4866-9aa0-32279501d542 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:42.936249 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca5cb127-91b3-4866-9aa0-32279501d542 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:42.940225 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca5cb127-91b3-4866-9aa0-32279501d542 service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:42.954435 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1216/2397] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:58:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 09 00:58:43.281907 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d53c1265-c6db-4f75-8c1a-0fb75e55d5f1 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:43.283444 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-d53c1265-c6db-4f75-8c1a-0fb75e55d5f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:58:43.295933 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d53c1265-c6db-4f75-8c1a-0fb75e55d5f1 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:43.301115 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a4e2c07-a25d-4e42-aff1-f1228ed8f176 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:43.301747 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a4e2c07-a25d-4e42-aff1-f1228ed8f176 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:43.302027 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a4e2c07-a25d-4e42-aff1-f1228ed8f176 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:43.302297 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a4e2c07-a25d-4e42-aff1-f1228ed8f176 admin admin] PATH_INFO: `/v3/projects/41cc12d5923847858268ee2c8be05164` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:43.303403 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a4e2c07-a25d-4e42-aff1-f1228ed8f176 admin admin] RBAC: Authorizing `identity:delete_project(project_id=41cc12d5923847858268ee2c8be05164)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:43.305552 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a4e2c07-a25d-4e42-aff1-f1228ed8f176 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:43.353252 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-7a4e2c07-a25d-4e42-aff1-f1228ed8f176 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '41cc12d5923847858268ee2c8be05164', 'request_id': 'req-7a4e2c07-a25d-4e42-aff1-f1228ed8f176'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:43.358767 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-7a4e2c07-a25d-4e42-aff1-f1228ed8f176 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '41cc12d5923847858268ee2c8be05164', 'request_id': 'req-7a4e2c07-a25d-4e42-aff1-f1228ed8f176'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:43.365455 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-7a4e2c07-a25d-4e42-aff1-f1228ed8f176 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '41cc12d5923847858268ee2c8be05164', 'request_id': 'req-7a4e2c07-a25d-4e42-aff1-f1228ed8f176'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:43.367986 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=93,keystone:DELETE=6,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:58:43.380387 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-7a4e2c07-a25d-4e42-aff1-f1228ed8f176 admin admin] The token cache is being invalidate because project 41cc12d5923847858268ee2c8be05164 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81453) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 00:58:43.380676 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-7a4e2c07-a25d-4e42-aff1-f1228ed8f176 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:43.381302 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-7a4e2c07-a25d-4e42-aff1-f1228ed8f176 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:43.382774 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1182/2398] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 00:58:43 2022] DELETE /identity/v3/projects/41cc12d5923847858268ee2c8be05164 => generated 0 bytes in 101 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 09 00:58:43.386678 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca5cb127-91b3-4866-9aa0-32279501d542 service neutron] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:43.399620 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca5cb127-91b3-4866-9aa0-32279501d542 service neutron] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:43.465319 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4429efa4-7f4d-40dc-bbb2-4e1e0669d401 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:43.465948 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4429efa4-7f4d-40dc-bbb2-4e1e0669d401 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:43.466395 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4429efa4-7f4d-40dc-bbb2-4e1e0669d401 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:43.466700 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4429efa4-7f4d-40dc-bbb2-4e1e0669d401 admin admin] PATH_INFO: `/v3/users/d8899e1ca4a941a2ac01d542801fc30d` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:43.467861 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4429efa4-7f4d-40dc-bbb2-4e1e0669d401 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d8899e1ca4a941a2ac01d542801fc30d)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:43.469894 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4429efa4-7f4d-40dc-bbb2-4e1e0669d401 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:43.524187 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-4429efa4-7f4d-40dc-bbb2-4e1e0669d401 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd8899e1ca4a941a2ac01d542801fc30d'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:43.532382 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-4429efa4-7f4d-40dc-bbb2-4e1e0669d401 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd8899e1ca4a941a2ac01d542801fc30d'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:43.533174 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-4429efa4-7f4d-40dc-bbb2-4e1e0669d401 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd8899e1ca4a941a2ac01d542801fc30d'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:43.533940 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-4429efa4-7f4d-40dc-bbb2-4e1e0669d401 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd8899e1ca4a941a2ac01d542801fc30d'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:43.542685 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-4429efa4-7f4d-40dc-bbb2-4e1e0669d401 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd8899e1ca4a941a2ac01d542801fc30d'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:43.544046 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:58:43.567638 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1217/2399] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 00:58:43 2022] DELETE /identity/v3/users/d8899e1ca4a941a2ac01d542801fc30d => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:58:43.696126 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a4e2c07-a25d-4e42-aff1-f1228ed8f176 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:43.697651 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-7a4e2c07-a25d-4e42-aff1-f1228ed8f176 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:58:43.712748 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a4e2c07-a25d-4e42-aff1-f1228ed8f176 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:43.786083 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4468ea30-0110-4786-a2db-49d1b932599e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:43.787569 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4468ea30-0110-4786-a2db-49d1b932599e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:43.788108 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4468ea30-0110-4786-a2db-49d1b932599e admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:43.788844 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4468ea30-0110-4786-a2db-49d1b932599e admin admin] PATH_INFO: `/v3/users/4a73ba10f87a49dfa3f9abca93819dea` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:43.790383 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4468ea30-0110-4786-a2db-49d1b932599e admin admin] RBAC: Authorizing `identity:delete_user(user_id=4a73ba10f87a49dfa3f9abca93819dea)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:43.793708 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4468ea30-0110-4786-a2db-49d1b932599e admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:43.863901 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-4468ea30-0110-4786-a2db-49d1b932599e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4a73ba10f87a49dfa3f9abca93819dea'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:43.884089 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-4468ea30-0110-4786-a2db-49d1b932599e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4a73ba10f87a49dfa3f9abca93819dea'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:43.884768 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-4468ea30-0110-4786-a2db-49d1b932599e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4a73ba10f87a49dfa3f9abca93819dea'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:43.896499 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-4468ea30-0110-4786-a2db-49d1b932599e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4a73ba10f87a49dfa3f9abca93819dea'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:43.903393 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4429efa4-7f4d-40dc-bbb2-4e1e0669d401 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:43.907514 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-4429efa4-7f4d-40dc-bbb2-4e1e0669d401 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:58:43.913646 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-4468ea30-0110-4786-a2db-49d1b932599e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4a73ba10f87a49dfa3f9abca93819dea'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:43.920253 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1183/2400] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 00:58:43 2022] DELETE /identity/v3/users/4a73ba10f87a49dfa3f9abca93819dea => generated 0 bytes in 224 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:58:43.924472 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c3b7c0a-78c5-46b9-92a7-e4504e4b7e2e None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:43.924756 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c3b7c0a-78c5-46b9-92a7-e4504e4b7e2e None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:43.925035 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c3b7c0a-78c5-46b9-92a7-e4504e4b7e2e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:43.954587 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4429efa4-7f4d-40dc-bbb2-4e1e0669d401 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:43.958070 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-8c3b7c0a-78c5-46b9-92a7-e4504e4b7e2e None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:58:43.959604 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-8c3b7c0a-78c5-46b9-92a7-e4504e4b7e2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:58:44.023698 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e925cc9d-0914-4275-9458-d1f3c884f6d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:44.024435 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e925cc9d-0914-4275-9458-d1f3c884f6d1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:44.024717 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e925cc9d-0914-4275-9458-d1f3c884f6d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:44.025015 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e925cc9d-0914-4275-9458-d1f3c884f6d1 admin admin] PATH_INFO: `/v3/projects/15814fc60c8c40188bbce5f0ce50db10` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:44.026204 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e925cc9d-0914-4275-9458-d1f3c884f6d1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=15814fc60c8c40188bbce5f0ce50db10)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:44.028138 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e925cc9d-0914-4275-9458-d1f3c884f6d1 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:44.035427 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1184/2401] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:58:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 09 00:58:44.046058 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c3b7c0a-78c5-46b9-92a7-e4504e4b7e2e None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:44.047890 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-8c3b7c0a-78c5-46b9-92a7-e4504e4b7e2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:58:44.066261 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:DELETE=11,keystone:SELECT=88,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:58:44.090063 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-e925cc9d-0914-4275-9458-d1f3c884f6d1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '15814fc60c8c40188bbce5f0ce50db10', 'request_id': 'req-e925cc9d-0914-4275-9458-d1f3c884f6d1'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:44.097514 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-e925cc9d-0914-4275-9458-d1f3c884f6d1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '15814fc60c8c40188bbce5f0ce50db10', 'request_id': 'req-e925cc9d-0914-4275-9458-d1f3c884f6d1'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:44.105703 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-e925cc9d-0914-4275-9458-d1f3c884f6d1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '15814fc60c8c40188bbce5f0ce50db10', 'request_id': 'req-e925cc9d-0914-4275-9458-d1f3c884f6d1'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:44.121717 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-e925cc9d-0914-4275-9458-d1f3c884f6d1 admin admin] The token cache is being invalidate because project 15814fc60c8c40188bbce5f0ce50db10 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 00:58:44.122544 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-e925cc9d-0914-4275-9458-d1f3c884f6d1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:44.123341 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-e925cc9d-0914-4275-9458-d1f3c884f6d1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:44.124850 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1218/2402] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 00:58:43 2022] DELETE /identity/v3/projects/15814fc60c8c40188bbce5f0ce50db10 => generated 0 bytes in 222 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 09 00:58:44.130178 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e925cc9d-0914-4275-9458-d1f3c884f6d1 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:44.131606 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-e925cc9d-0914-4275-9458-d1f3c884f6d1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:58:44.134011 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c3b7c0a-78c5-46b9-92a7-e4504e4b7e2e None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:44.234717 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e925cc9d-0914-4275-9458-d1f3c884f6d1 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:44.242270 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc9ef111-0bbd-4c86-9f29-891557265dbb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:44.243110 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9ef111-0bbd-4c86-9f29-891557265dbb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:44.243355 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-974631b4-84b3-4384-b0ed-bc8c5bb96284 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:44.243522 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9ef111-0bbd-4c86-9f29-891557265dbb admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:44.243994 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9ef111-0bbd-4c86-9f29-891557265dbb admin admin] PATH_INFO: `/v3/users/e287fcf5c6b5434fa9f9e86c51fce919` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:44.244154 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974631b4-84b3-4384-b0ed-bc8c5bb96284 service neutron] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:44.244537 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974631b4-84b3-4384-b0ed-bc8c5bb96284 service neutron] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:44.244706 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974631b4-84b3-4384-b0ed-bc8c5bb96284 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:44.245240 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc9ef111-0bbd-4c86-9f29-891557265dbb admin admin] RBAC: Authorizing `identity:delete_user(user_id=e287fcf5c6b5434fa9f9e86c51fce919)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:44.246615 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-974631b4-84b3-4384-b0ed-bc8c5bb96284 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:58:44.247276 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc9ef111-0bbd-4c86-9f29-891557265dbb admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:44.267318 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-974631b4-84b3-4384-b0ed-bc8c5bb96284 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:44.269984 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-974631b4-84b3-4384-b0ed-bc8c5bb96284 service neutron] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:44.271046 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:58:44.285022 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1185/2403] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:58:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 239 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 09 00:58:44.319123 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-bc9ef111-0bbd-4c86-9f29-891557265dbb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e287fcf5c6b5434fa9f9e86c51fce919'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:44.326906 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-bc9ef111-0bbd-4c86-9f29-891557265dbb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e287fcf5c6b5434fa9f9e86c51fce919'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:44.327891 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-bc9ef111-0bbd-4c86-9f29-891557265dbb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e287fcf5c6b5434fa9f9e86c51fce919'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:44.328650 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-bc9ef111-0bbd-4c86-9f29-891557265dbb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e287fcf5c6b5434fa9f9e86c51fce919'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:44.338846 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-bc9ef111-0bbd-4c86-9f29-891557265dbb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e287fcf5c6b5434fa9f9e86c51fce919'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:44.362630 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1219/2404] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 00:58:44 2022] DELETE /identity/v3/users/e287fcf5c6b5434fa9f9e86c51fce919 => generated 0 bytes in 233 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 09 00:58:44.729256 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-974631b4-84b3-4384-b0ed-bc8c5bb96284 service neutron] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:44.730787 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-974631b4-84b3-4384-b0ed-bc8c5bb96284 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:58:44.737402 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc9ef111-0bbd-4c86-9f29-891557265dbb admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:44.738847 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-bc9ef111-0bbd-4c86-9f29-891557265dbb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:58:44.750096 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-974631b4-84b3-4384-b0ed-bc8c5bb96284 service neutron] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:44.812619 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a067592d-9228-4ad9-bc7e-62d18ee156ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:44.813307 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a067592d-9228-4ad9-bc7e-62d18ee156ac admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:44.813660 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a067592d-9228-4ad9-bc7e-62d18ee156ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:44.814004 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a067592d-9228-4ad9-bc7e-62d18ee156ac admin admin] PATH_INFO: `/v3/projects/855bbf7d882d4e88a30ff3b1d83dd4af` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:44.815286 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a067592d-9228-4ad9-bc7e-62d18ee156ac admin admin] RBAC: Authorizing `identity:delete_project(project_id=855bbf7d882d4e88a30ff3b1d83dd4af)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:44.817322 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a067592d-9228-4ad9-bc7e-62d18ee156ac admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:44.821382 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc9ef111-0bbd-4c86-9f29-891557265dbb admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:44.825948 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e09f1283-87ba-4cee-bfc0-a7f07a964e57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:44.826563 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e09f1283-87ba-4cee-bfc0-a7f07a964e57 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:44.826846 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e09f1283-87ba-4cee-bfc0-a7f07a964e57 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:44.827114 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e09f1283-87ba-4cee-bfc0-a7f07a964e57 admin admin] PATH_INFO: `/v3/projects/508281ed1de64e0ebaecd337b7d039f2` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:44.828302 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e09f1283-87ba-4cee-bfc0-a7f07a964e57 admin admin] RBAC: Authorizing `identity:delete_project(project_id=508281ed1de64e0ebaecd337b7d039f2)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:44.830761 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e09f1283-87ba-4cee-bfc0-a7f07a964e57 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:44.869687 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-a067592d-9228-4ad9-bc7e-62d18ee156ac admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '855bbf7d882d4e88a30ff3b1d83dd4af', 'request_id': 'req-a067592d-9228-4ad9-bc7e-62d18ee156ac'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:44.876364 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-a067592d-9228-4ad9-bc7e-62d18ee156ac admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '855bbf7d882d4e88a30ff3b1d83dd4af', 'request_id': 'req-a067592d-9228-4ad9-bc7e-62d18ee156ac'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:44.880157 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-e09f1283-87ba-4cee-bfc0-a7f07a964e57 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '508281ed1de64e0ebaecd337b7d039f2', 'request_id': 'req-e09f1283-87ba-4cee-bfc0-a7f07a964e57'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:44.883074 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-a067592d-9228-4ad9-bc7e-62d18ee156ac admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '855bbf7d882d4e88a30ff3b1d83dd4af', 'request_id': 'req-a067592d-9228-4ad9-bc7e-62d18ee156ac'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:44.886683 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-e09f1283-87ba-4cee-bfc0-a7f07a964e57 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '508281ed1de64e0ebaecd337b7d039f2', 'request_id': 'req-e09f1283-87ba-4cee-bfc0-a7f07a964e57'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:44.894874 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-e09f1283-87ba-4cee-bfc0-a7f07a964e57 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '508281ed1de64e0ebaecd337b7d039f2', 'request_id': 'req-e09f1283-87ba-4cee-bfc0-a7f07a964e57'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:44.899359 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-a067592d-9228-4ad9-bc7e-62d18ee156ac admin admin] The token cache is being invalidate because project 855bbf7d882d4e88a30ff3b1d83dd4af was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81453) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 00:58:44.899822 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-a067592d-9228-4ad9-bc7e-62d18ee156ac admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:44.900661 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-a067592d-9228-4ad9-bc7e-62d18ee156ac admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:44.902708 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1186/2405] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 00:58:44 2022] DELETE /identity/v3/projects/855bbf7d882d4e88a30ff3b1d83dd4af => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:58:44.907532 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffbef688-9c26-4031-bfe8-d395026ab503 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:44.907847 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffbef688-9c26-4031-bfe8-d395026ab503 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:44.908821 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffbef688-9c26-4031-bfe8-d395026ab503 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:44.912231 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-e09f1283-87ba-4cee-bfc0-a7f07a964e57 admin admin] The token cache is being invalidate because project 508281ed1de64e0ebaecd337b7d039f2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 00:58:44.912692 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-e09f1283-87ba-4cee-bfc0-a7f07a964e57 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:44.913209 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-e09f1283-87ba-4cee-bfc0-a7f07a964e57 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:44.915020 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1220/2406] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 00:58:44 2022] DELETE /identity/v3/projects/508281ed1de64e0ebaecd337b7d039f2 => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:58:44.918644 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:58:44.919752 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5db34e9-93f4-46b9-8025-4ed93da2f311 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:44.920017 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5db34e9-93f4-46b9-8025-4ed93da2f311 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:44.920597 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5db34e9-93f4-46b9-8025-4ed93da2f311 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:44.935932 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-ffbef688-9c26-4031-bfe8-d395026ab503 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:58:44.937247 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-ffbef688-9c26-4031-bfe8-d395026ab503 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:58:44.942041 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-f5db34e9-93f4-46b9-8025-4ed93da2f311 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:58:44.943216 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-f5db34e9-93f4-46b9-8025-4ed93da2f311 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:58:44.960358 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=82,keystone:DELETE=16,keystone:INSERT=2 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:58:45.007194 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1187/2407] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:58:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 09 00:58:45.015036 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ff71ea-7c90-4a1d-87ab-34f9f8b08e52 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:45.015444 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ff71ea-7c90-4a1d-87ab-34f9f8b08e52 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:45.015752 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ff71ea-7c90-4a1d-87ab-34f9f8b08e52 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:45.017472 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1221/2408] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:58:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 09 00:58:45.025325 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cdc4e5e-2e3d-4d9c-810b-cfe0fffdcc03 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:45.025678 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cdc4e5e-2e3d-4d9c-810b-cfe0fffdcc03 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:45.026014 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cdc4e5e-2e3d-4d9c-810b-cfe0fffdcc03 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:45.036951 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-10ff71ea-7c90-4a1d-87ab-34f9f8b08e52 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:58:45.038154 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-10ff71ea-7c90-4a1d-87ab-34f9f8b08e52 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:58:45.045249 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-3cdc4e5e-2e3d-4d9c-810b-cfe0fffdcc03 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:58:45.045908 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1188/2409] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:58:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 09 00:58:45.046394 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-3cdc4e5e-2e3d-4d9c-810b-cfe0fffdcc03 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:58:45.050036 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10ff71ea-7c90-4a1d-87ab-34f9f8b08e52 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:45.054888 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1222/2410] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:58:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 09 00:58:45.059297 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cdc4e5e-2e3d-4d9c-810b-cfe0fffdcc03 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:45.059573 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10ff71ea-7c90-4a1d-87ab-34f9f8b08e52 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:45.064922 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c310da36-5147-4ac1-8920-fcc8f3ac2a02 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:45.065521 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c310da36-5147-4ac1-8920-fcc8f3ac2a02 admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:45.065837 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c310da36-5147-4ac1-8920-fcc8f3ac2a02 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:45.066108 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c310da36-5147-4ac1-8920-fcc8f3ac2a02 admin admin] PATH_INFO: `/v3/domains` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:45.066744 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c310da36-5147-4ac1-8920-fcc8f3ac2a02 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:58:45.067044 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c310da36-5147-4ac1-8920-fcc8f3ac2a02 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:45.068654 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c310da36-5147-4ac1-8920-fcc8f3ac2a02 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:45.069205 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cdc4e5e-2e3d-4d9c-810b-cfe0fffdcc03 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:45.074165 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8ebf9d6-499e-4998-845f-305df27dbca9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:45.074752 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ebf9d6-499e-4998-845f-305df27dbca9 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:45.075027 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ebf9d6-499e-4998-845f-305df27dbca9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:45.075296 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ebf9d6-499e-4998-845f-305df27dbca9 admin admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:45.075975 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8ebf9d6-499e-4998-845f-305df27dbca9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:58:45.076287 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8ebf9d6-499e-4998-845f-305df27dbca9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:45.077837 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8ebf9d6-499e-4998-845f-305df27dbca9 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:45.087108 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1189/2411] 10.222.0.33 () {58 vars in 1153 bytes} [Sun Oct 9 00:58:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:58:45.088178 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c310da36-5147-4ac1-8920-fcc8f3ac2a02 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:45.094619 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1223/2412] 10.222.0.33 () {58 vars in 1153 bytes} [Sun Oct 9 00:58:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 00:58:45.095988 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8ebf9d6-499e-4998-845f-305df27dbca9 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:45.098408 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c310da36-5147-4ac1-8920-fcc8f3ac2a02 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:45.103704 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a7f4d14-81d6-496c-82b7-7ddf0303c495 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:45.104592 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a7f4d14-81d6-496c-82b7-7ddf0303c495 admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:45.104740 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a7f4d14-81d6-496c-82b7-7ddf0303c495 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:45.104985 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a7f4d14-81d6-496c-82b7-7ddf0303c495 admin admin] PATH_INFO: `/v3/projects` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:45.105624 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a7f4d14-81d6-496c-82b7-7ddf0303c495 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:45.106296 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8ebf9d6-499e-4998-845f-305df27dbca9 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:45.107627 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a7f4d14-81d6-496c-82b7-7ddf0303c495 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:45.111666 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-499a5f13-fb89-4ac7-99aa-99452fc7078c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:45.112368 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-499a5f13-fb89-4ac7-99aa-99452fc7078c admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:45.112730 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-499a5f13-fb89-4ac7-99aa-99452fc7078c admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:45.113086 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-499a5f13-fb89-4ac7-99aa-99452fc7078c admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:45.113921 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-499a5f13-fb89-4ac7-99aa-99452fc7078c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:45.116083 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-499a5f13-fb89-4ac7-99aa-99452fc7078c admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:45.136493 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1190/2413] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 00:58:45 2022] POST /identity/v3/projects => generated 373 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 09 00:58:45.141250 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a7f4d14-81d6-496c-82b7-7ddf0303c495 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:45.146188 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1224/2414] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 00:58:45 2022] POST /identity/v3/projects => generated 371 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 09 00:58:45.150368 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-499a5f13-fb89-4ac7-99aa-99452fc7078c admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:45.208571 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a7f4d14-81d6-496c-82b7-7ddf0303c495 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:45.213317 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea12983e-854a-497d-b089-a722d64979e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:45.214040 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea12983e-854a-497d-b089-a722d64979e6 admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:45.214320 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea12983e-854a-497d-b089-a722d64979e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:45.214434 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-499a5f13-fb89-4ac7-99aa-99452fc7078c admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:45.214619 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea12983e-854a-497d-b089-a722d64979e6 admin admin] PATH_INFO: `/v3/users` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:45.215432 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea12983e-854a-497d-b089-a722d64979e6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:45.217569 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea12983e-854a-497d-b089-a722d64979e6 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:45.219244 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ebbaa8d-c3fa-4b8e-a67a-723db6846dba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:45.219883 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ebbaa8d-c3fa-4b8e-a67a-723db6846dba admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:45.220166 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ebbaa8d-c3fa-4b8e-a67a-723db6846dba admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:45.220480 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ebbaa8d-c3fa-4b8e-a67a-723db6846dba admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:45.221093 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ebbaa8d-c3fa-4b8e-a67a-723db6846dba admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:45.223006 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ebbaa8d-c3fa-4b8e-a67a-723db6846dba admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:45.240315 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1191/2415] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:58:45 2022] POST /identity/v3/users => generated 287 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 09 00:58:45.244473 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea12983e-854a-497d-b089-a722d64979e6 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:45.247606 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1225/2416] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:58:45 2022] POST /identity/v3/users => generated 286 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 09 00:58:45.249110 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ebbaa8d-c3fa-4b8e-a67a-723db6846dba admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:45.255429 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea12983e-854a-497d-b089-a722d64979e6 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:45.260652 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ebbaa8d-c3fa-4b8e-a67a-723db6846dba admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:45.260794 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9290e715-fc3a-4b54-bf86-5fce8e5b1ecf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:45.261258 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9290e715-fc3a-4b54-bf86-5fce8e5b1ecf admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:45.261568 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9290e715-fc3a-4b54-bf86-5fce8e5b1ecf admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:45.261846 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9290e715-fc3a-4b54-bf86-5fce8e5b1ecf admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:45.262479 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9290e715-fc3a-4b54-bf86-5fce8e5b1ecf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:45.264103 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9290e715-fc3a-4b54-bf86-5fce8e5b1ecf admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:45.264214 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d104c6ea-a2ab-4e8e-a182-0c8b6cd02270 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:45.264734 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d104c6ea-a2ab-4e8e-a182-0c8b6cd02270 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:45.265019 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d104c6ea-a2ab-4e8e-a182-0c8b6cd02270 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:45.265452 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d104c6ea-a2ab-4e8e-a182-0c8b6cd02270 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:45.266093 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d104c6ea-a2ab-4e8e-a182-0c8b6cd02270 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:45.268657 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d104c6ea-a2ab-4e8e-a182-0c8b6cd02270 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:45.276185 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1192/2417] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:58:45 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:58:45.279648 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9290e715-fc3a-4b54-bf86-5fce8e5b1ecf admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:45.282799 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1226/2418] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:58:45 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:58:45.286925 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d104c6ea-a2ab-4e8e-a182-0c8b6cd02270 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:45.294988 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9290e715-fc3a-4b54-bf86-5fce8e5b1ecf admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:45.297234 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d104c6ea-a2ab-4e8e-a182-0c8b6cd02270 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:45.300864 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83c5b079-0b75-4624-be75-b153182e4647 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:45.301552 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c5b079-0b75-4624-be75-b153182e4647 admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:45.301836 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c5b079-0b75-4624-be75-b153182e4647 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:45.302114 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c5b079-0b75-4624-be75-b153182e4647 admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:45.302817 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83c5b079-0b75-4624-be75-b153182e4647 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:45.304500 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83c5b079-0b75-4624-be75-b153182e4647 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:45.304811 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdaae762-8a22-49a5-b7f7-9a31e1e73d5c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:45.305443 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdaae762-8a22-49a5-b7f7-9a31e1e73d5c admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:45.305719 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdaae762-8a22-49a5-b7f7-9a31e1e73d5c admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:45.305991 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdaae762-8a22-49a5-b7f7-9a31e1e73d5c admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:45.306623 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdaae762-8a22-49a5-b7f7-9a31e1e73d5c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:45.308523 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdaae762-8a22-49a5-b7f7-9a31e1e73d5c admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:45.319637 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1193/2419] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:58:45 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:58:45.323363 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1227/2420] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:58:45 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 09 00:58:45.324543 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdaae762-8a22-49a5-b7f7-9a31e1e73d5c admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:45.324682 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83c5b079-0b75-4624-be75-b153182e4647 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:45.335398 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83c5b079-0b75-4624-be75-b153182e4647 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:45.337442 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdaae762-8a22-49a5-b7f7-9a31e1e73d5c admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:45.340296 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c44fd4ce-0331-4810-a7b0-495cf019c317 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:45.340945 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c44fd4ce-0331-4810-a7b0-495cf019c317 admin admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:45.341225 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c44fd4ce-0331-4810-a7b0-495cf019c317 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:45.341932 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c44fd4ce-0331-4810-a7b0-495cf019c317 admin admin] PATH_INFO: `/v3/projects/03feb5df12294d1291aaabcec61c9150/users/6f344779afd042099cd427c29422ac36/roles/588016933bab436d8b7c3ac76827356d` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:45.344408 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e32b3c29-f367-441f-a6f8-f59896eea4de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:45.346002 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e32b3c29-f367-441f-a6f8-f59896eea4de admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:45.346285 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e32b3c29-f367-441f-a6f8-f59896eea4de admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:45.346565 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e32b3c29-f367-441f-a6f8-f59896eea4de admin admin] PATH_INFO: `/v3/projects/5edae5583d2e4a4688bef23f3a14eda4/users/adcbf206828d49b9bf2ae720f71068bc/roles/588016933bab436d8b7c3ac76827356d` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:45.358975 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c44fd4ce-0331-4810-a7b0-495cf019c317 admin admin] RBAC: Authorizing `identity:create_grant(project_id=03feb5df12294d1291aaabcec61c9150, user_id=6f344779afd042099cd427c29422ac36, role_id=588016933bab436d8b7c3ac76827356d)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:45.362512 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c44fd4ce-0331-4810-a7b0-495cf019c317 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:45.364852 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e32b3c29-f367-441f-a6f8-f59896eea4de admin admin] RBAC: Authorizing `identity:create_grant(project_id=5edae5583d2e4a4688bef23f3a14eda4, user_id=adcbf206828d49b9bf2ae720f71068bc, role_id=588016933bab436d8b7c3ac76827356d)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:45.368408 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e32b3c29-f367-441f-a6f8-f59896eea4de admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:45.379680 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1194/2421] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:58:45 2022] PUT /identity/v3/projects/03feb5df12294d1291aaabcec61c9150/users/6f344779afd042099cd427c29422ac36/roles/588016933bab436d8b7c3ac76827356d => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:58:45.389660 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7900975-5e88-4eea-b686-435f1b26e50c None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:45.389962 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7900975-5e88-4eea-b686-435f1b26e50c None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:45.390219 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7900975-5e88-4eea-b686-435f1b26e50c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:45.390648 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1228/2422] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:58:45 2022] PUT /identity/v3/projects/5edae5583d2e4a4688bef23f3a14eda4/users/adcbf206828d49b9bf2ae720f71068bc/roles/588016933bab436d8b7c3ac76827356d => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:58:45.395512 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cab635e-96f6-4309-b260-c43d52e23075 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:45.395879 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cab635e-96f6-4309-b260-c43d52e23075 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:45.396168 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cab635e-96f6-4309-b260-c43d52e23075 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:45.399239 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:58:45.423380 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-c7900975-5e88-4eea-b686-435f1b26e50c None None] MFA Rules not processed for user `6f344779afd042099cd427c29422ac36`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:58:45.425310 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-c7900975-5e88-4eea-b686-435f1b26e50c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:58:45.430763 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-5cab635e-96f6-4309-b260-c43d52e23075 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:58:45.431976 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-5cab635e-96f6-4309-b260-c43d52e23075 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:58:45.452250 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:58:45.495826 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1195/2423] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:58:45 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:58:45.503639 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1229/2424] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:58:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:58:45.508065 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7900975-5e88-4eea-b686-435f1b26e50c None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:45.509470 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-c7900975-5e88-4eea-b686-435f1b26e50c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:58:45.514739 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cab635e-96f6-4309-b260-c43d52e23075 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:45.516136 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-5cab635e-96f6-4309-b260-c43d52e23075 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:58:45.587775 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7900975-5e88-4eea-b686-435f1b26e50c None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:45.596271 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cab635e-96f6-4309-b260-c43d52e23075 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:45.607044 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f59ad4c-6c83-4734-abd4-7a20242cde57 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3f48335668a4e8f934ac11ff66cd308', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:45.607826 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f59ad4c-6c83-4734-abd4-7a20242cde57 service cinder] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:45.608177 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f59ad4c-6c83-4734-abd4-7a20242cde57 service cinder] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:45.608571 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f59ad4c-6c83-4734-abd4-7a20242cde57 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:45.612346 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dec86171-2f2f-427a-8219-f2292bfa1284 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:45.612960 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec86171-2f2f-427a-8219-f2292bfa1284 service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:45.613563 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec86171-2f2f-427a-8219-f2292bfa1284 service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:45.614148 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec86171-2f2f-427a-8219-f2292bfa1284 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:45.620304 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f59ad4c-6c83-4734-abd4-7a20242cde57 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:45.622129 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f59ad4c-6c83-4734-abd4-7a20242cde57 service cinder] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:45.626079 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dec86171-2f2f-427a-8219-f2292bfa1284 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:45.628906 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dec86171-2f2f-427a-8219-f2292bfa1284 service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:45.632884 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1196/2425] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:58:45 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 125 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:58:45.641741 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1230/2426] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:58:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 127 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 09 00:58:45.663762 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f59ad4c-6c83-4734-abd4-7a20242cde57 service cinder] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:45.678799 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f59ad4c-6c83-4734-abd4-7a20242cde57 service cinder] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:45.683941 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-505fe2e2-3120-4b3d-a15f-534fcfa40779 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:45.684598 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-505fe2e2-3120-4b3d-a15f-534fcfa40779 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:45.684891 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-505fe2e2-3120-4b3d-a15f-534fcfa40779 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:45.685162 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-505fe2e2-3120-4b3d-a15f-534fcfa40779 admin admin] PATH_INFO: `/v3/users/6f344779afd042099cd427c29422ac36` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:45.686405 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-505fe2e2-3120-4b3d-a15f-534fcfa40779 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6f344779afd042099cd427c29422ac36)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:45.688380 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-505fe2e2-3120-4b3d-a15f-534fcfa40779 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:45.751498 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-505fe2e2-3120-4b3d-a15f-534fcfa40779 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6f344779afd042099cd427c29422ac36'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:45.759830 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-505fe2e2-3120-4b3d-a15f-534fcfa40779 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6f344779afd042099cd427c29422ac36'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:45.760717 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-505fe2e2-3120-4b3d-a15f-534fcfa40779 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6f344779afd042099cd427c29422ac36'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:45.771083 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-505fe2e2-3120-4b3d-a15f-534fcfa40779 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6f344779afd042099cd427c29422ac36'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:45.799276 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-505fe2e2-3120-4b3d-a15f-534fcfa40779 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6f344779afd042099cd427c29422ac36'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:45.805817 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1197/2427] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 00:58:45 2022] DELETE /identity/v3/users/6f344779afd042099cd427c29422ac36 => generated 0 bytes in 143 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:58:45.810436 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dbd5bd7-3068-48d8-9943-50df30b86dfc None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:45.812116 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dbd5bd7-3068-48d8-9943-50df30b86dfc None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:45.812116 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dbd5bd7-3068-48d8-9943-50df30b86dfc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:45.833959 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-5dbd5bd7-3068-48d8-9943-50df30b86dfc None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:58:45.835288 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-5dbd5bd7-3068-48d8-9943-50df30b86dfc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:58:45.912720 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1231/2428] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:58:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:58:45.921956 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-505fe2e2-3120-4b3d-a15f-534fcfa40779 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:45.924183 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-505fe2e2-3120-4b3d-a15f-534fcfa40779 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:58:45.936312 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:58:45.992387 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-505fe2e2-3120-4b3d-a15f-534fcfa40779 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:45.996144 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2ff3932-b347-4db7-95d1-58e2f472c478 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:45.996763 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2ff3932-b347-4db7-95d1-58e2f472c478 service neutron] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:45.997091 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2ff3932-b347-4db7-95d1-58e2f472c478 service neutron] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:45.997368 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2ff3932-b347-4db7-95d1-58e2f472c478 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:46.009161 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2ff3932-b347-4db7-95d1-58e2f472c478 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:46.010601 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2ff3932-b347-4db7-95d1-58e2f472c478 service neutron] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:46.022532 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1198/2429] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:58:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:58:46.375908 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dbd5bd7-3068-48d8-9943-50df30b86dfc None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:46.377966 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-5dbd5bd7-3068-48d8-9943-50df30b86dfc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:58:46.394152 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dbd5bd7-3068-48d8-9943-50df30b86dfc None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:46.400875 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45add304-7b94-454f-bb65-97a4ce855a86 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:46.401505 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45add304-7b94-454f-bb65-97a4ce855a86 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:46.401783 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45add304-7b94-454f-bb65-97a4ce855a86 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:46.402054 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45add304-7b94-454f-bb65-97a4ce855a86 admin admin] PATH_INFO: `/v3/projects/03feb5df12294d1291aaabcec61c9150` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:46.403792 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45add304-7b94-454f-bb65-97a4ce855a86 admin admin] RBAC: Authorizing `identity:delete_project(project_id=03feb5df12294d1291aaabcec61c9150)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:46.406307 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45add304-7b94-454f-bb65-97a4ce855a86 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:46.464713 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-45add304-7b94-454f-bb65-97a4ce855a86 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '03feb5df12294d1291aaabcec61c9150', 'request_id': 'req-45add304-7b94-454f-bb65-97a4ce855a86'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:46.481687 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-45add304-7b94-454f-bb65-97a4ce855a86 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '03feb5df12294d1291aaabcec61c9150', 'request_id': 'req-45add304-7b94-454f-bb65-97a4ce855a86'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:46.488060 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=94,keystone:DELETE=6 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:58:46.498291 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-45add304-7b94-454f-bb65-97a4ce855a86 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '03feb5df12294d1291aaabcec61c9150', 'request_id': 'req-45add304-7b94-454f-bb65-97a4ce855a86'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:46.520331 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-45add304-7b94-454f-bb65-97a4ce855a86 admin admin] The token cache is being invalidate because project 03feb5df12294d1291aaabcec61c9150 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 00:58:46.520679 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-45add304-7b94-454f-bb65-97a4ce855a86 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:46.521824 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-45add304-7b94-454f-bb65-97a4ce855a86 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:46.540625 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1232/2430] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 00:58:46 2022] DELETE /identity/v3/projects/03feb5df12294d1291aaabcec61c9150 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:58:46.707624 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8457d08f-ed84-47d1-be60-4d8a4bc233da None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:46.708327 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8457d08f-ed84-47d1-be60-4d8a4bc233da None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:46.708784 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8457d08f-ed84-47d1-be60-4d8a4bc233da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:46.756998 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-8457d08f-ed84-47d1-be60-4d8a4bc233da None None] MFA Rules not processed for user `adcbf206828d49b9bf2ae720f71068bc`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:58:46.758611 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-8457d08f-ed84-47d1-be60-4d8a4bc233da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:58:46.834281 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1199/2431] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:58:46 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 09 00:58:46.843116 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45add304-7b94-454f-bb65-97a4ce855a86 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:46.844827 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-45add304-7b94-454f-bb65-97a4ce855a86 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:58:46.917826 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45add304-7b94-454f-bb65-97a4ce855a86 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:46.921582 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e23777d-17a9-4d0b-b621-695831512d22 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3f48335668a4e8f934ac11ff66cd308', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:46.922218 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e23777d-17a9-4d0b-b621-695831512d22 service cinder] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:46.922504 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e23777d-17a9-4d0b-b621-695831512d22 service cinder] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:46.922776 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e23777d-17a9-4d0b-b621-695831512d22 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:46.931425 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e23777d-17a9-4d0b-b621-695831512d22 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:46.932884 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e23777d-17a9-4d0b-b621-695831512d22 service cinder] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:46.942009 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1233/2432] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:58:46 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 09 00:58:48.294907 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07a61918-2285-449c-951b-877f0a70fbde None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:48.295426 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07a61918-2285-449c-951b-877f0a70fbde None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:48.295522 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07a61918-2285-449c-951b-877f0a70fbde None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:48.314810 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-07a61918-2285-449c-951b-877f0a70fbde None None] MFA Rules not processed for user `adcbf206828d49b9bf2ae720f71068bc`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:58:48.316072 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-07a61918-2285-449c-951b-877f0a70fbde None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:58:48.328350 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1200/2433] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:58:48 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:58:50.710075 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e23777d-17a9-4d0b-b621-695831512d22 service cinder] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:50.711478 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-1e23777d-17a9-4d0b-b621-695831512d22 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:58:50.775545 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e23777d-17a9-4d0b-b621-695831512d22 service cinder] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:50.780270 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63b039ff-fd52-426f-b02b-bbba3828a0aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:50.780884 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b039ff-fd52-426f-b02b-bbba3828a0aa admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:50.781163 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b039ff-fd52-426f-b02b-bbba3828a0aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:50.781434 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b039ff-fd52-426f-b02b-bbba3828a0aa admin admin] PATH_INFO: `/v3/users/b5ee382f8e4642589c39c6f49d7615d2` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:50.782553 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63b039ff-fd52-426f-b02b-bbba3828a0aa admin admin] RBAC: Authorizing `identity:delete_user(user_id=b5ee382f8e4642589c39c6f49d7615d2)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:50.784499 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63b039ff-fd52-426f-b02b-bbba3828a0aa admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:50.829641 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-63b039ff-fd52-426f-b02b-bbba3828a0aa admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b5ee382f8e4642589c39c6f49d7615d2'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:50.836542 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-63b039ff-fd52-426f-b02b-bbba3828a0aa admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b5ee382f8e4642589c39c6f49d7615d2'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:50.837187 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-63b039ff-fd52-426f-b02b-bbba3828a0aa admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b5ee382f8e4642589c39c6f49d7615d2'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:50.837809 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-63b039ff-fd52-426f-b02b-bbba3828a0aa admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b5ee382f8e4642589c39c6f49d7615d2'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:50.844717 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-63b039ff-fd52-426f-b02b-bbba3828a0aa admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b5ee382f8e4642589c39c6f49d7615d2'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:50.862769 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1234/2434] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 00:58:50 2022] DELETE /identity/v3/users/b5ee382f8e4642589c39c6f49d7615d2 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:58:50.867339 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f56d575-1a66-4297-b68b-32d002304bfc None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:50.867833 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f56d575-1a66-4297-b68b-32d002304bfc None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:50.867969 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f56d575-1a66-4297-b68b-32d002304bfc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:50.887106 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-8f56d575-1a66-4297-b68b-32d002304bfc None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:58:50.888365 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-8f56d575-1a66-4297-b68b-32d002304bfc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:58:50.944725 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1201/2435] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:58:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:58:50.954188 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63b039ff-fd52-426f-b02b-bbba3828a0aa admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:50.955586 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-63b039ff-fd52-426f-b02b-bbba3828a0aa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:58:50.969183 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=87,keystone:INSERT=2,keystone:DELETE=11 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:58:51.013533 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63b039ff-fd52-426f-b02b-bbba3828a0aa admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:51.017115 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eedf74b-31d4-49cf-81d5-be165dfa0098 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:51.017682 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eedf74b-31d4-49cf-81d5-be165dfa0098 service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:51.017958 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eedf74b-31d4-49cf-81d5-be165dfa0098 service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:51.018227 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eedf74b-31d4-49cf-81d5-be165dfa0098 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:51.026857 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eedf74b-31d4-49cf-81d5-be165dfa0098 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:51.028297 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eedf74b-31d4-49cf-81d5-be165dfa0098 service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:51.038227 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1235/2436] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:58:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:58:51.403901 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f56d575-1a66-4297-b68b-32d002304bfc None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:58:51.405664 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-8f56d575-1a66-4297-b68b-32d002304bfc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:58:51.428928 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f56d575-1a66-4297-b68b-32d002304bfc None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:58:51.434115 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1caa318-cd9b-4ead-b6a9-2d0661691800 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:58:51.434993 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1caa318-cd9b-4ead-b6a9-2d0661691800 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:58:51.435434 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1caa318-cd9b-4ead-b6a9-2d0661691800 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:58:51.435992 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1caa318-cd9b-4ead-b6a9-2d0661691800 admin admin] PATH_INFO: `/v3/projects/ebbbad64bcf24f6c9667d7a5282a0f00` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:58:51.437524 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1caa318-cd9b-4ead-b6a9-2d0661691800 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ebbbad64bcf24f6c9667d7a5282a0f00)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:58:51.439890 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1caa318-cd9b-4ead-b6a9-2d0661691800 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:58:51.476228 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:58:51.489181 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-d1caa318-cd9b-4ead-b6a9-2d0661691800 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ebbbad64bcf24f6c9667d7a5282a0f00', 'request_id': 'req-d1caa318-cd9b-4ead-b6a9-2d0661691800'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:51.494624 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-d1caa318-cd9b-4ead-b6a9-2d0661691800 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ebbbad64bcf24f6c9667d7a5282a0f00', 'request_id': 'req-d1caa318-cd9b-4ead-b6a9-2d0661691800'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:51.502116 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-d1caa318-cd9b-4ead-b6a9-2d0661691800 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ebbbad64bcf24f6c9667d7a5282a0f00', 'request_id': 'req-d1caa318-cd9b-4ead-b6a9-2d0661691800'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:51.515285 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-d1caa318-cd9b-4ead-b6a9-2d0661691800 admin admin] The token cache is being invalidate because project ebbbad64bcf24f6c9667d7a5282a0f00 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81453) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 00:58:51.515576 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-d1caa318-cd9b-4ead-b6a9-2d0661691800 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:51.517696 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-d1caa318-cd9b-4ead-b6a9-2d0661691800 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:58:51.519254 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1202/2437] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 00:58:51 2022] DELETE /identity/v3/projects/ebbbad64bcf24f6c9667d7a5282a0f00 => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:59:01.034152 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=21 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:59:01.512207 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=13,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:59:07.091074 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eedf74b-31d4-49cf-81d5-be165dfa0098 service neutron] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:59:07.100226 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eedf74b-31d4-49cf-81d5-be165dfa0098 service neutron] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:59:07.147616 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7be060fc-6075-46b3-b7b0-5a31270e66b9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:59:07.148155 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be060fc-6075-46b3-b7b0-5a31270e66b9 service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:59:07.148437 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be060fc-6075-46b3-b7b0-5a31270e66b9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:59:07.148713 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be060fc-6075-46b3-b7b0-5a31270e66b9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:59:07.150447 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-7be060fc-6075-46b3-b7b0-5a31270e66b9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:59:07.212666 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7be060fc-6075-46b3-b7b0-5a31270e66b9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:59:07.214817 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7be060fc-6075-46b3-b7b0-5a31270e66b9 service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:59:07.225142 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1236/2438] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:59:07 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 135 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 00:59:09.867457 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1caa318-cd9b-4ead-b6a9-2d0661691800 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:59:09.869129 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-d1caa318-cd9b-4ead-b6a9-2d0661691800 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:59:09.968659 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1caa318-cd9b-4ead-b6a9-2d0661691800 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:59:09.974035 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eb39afe-cb3c-42b3-bb1a-32456c2135c2 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd25e240cb93643ed8122ccac9586936f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:59:09.975159 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb39afe-cb3c-42b3-bb1a-32456c2135c2 service placement] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:59:09.975159 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb39afe-cb3c-42b3-bb1a-32456c2135c2 service placement] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:59:09.975159 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eb39afe-cb3c-42b3-bb1a-32456c2135c2 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:59:09.977243 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-9eb39afe-cb3c-42b3-bb1a-32456c2135c2 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:59:09.995263 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eb39afe-cb3c-42b3-bb1a-32456c2135c2 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:59:09.998497 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eb39afe-cb3c-42b3-bb1a-32456c2135c2 service placement] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:59:10.010025 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1203/2439] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:59:09 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 144 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 09 00:59:14.153306 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-394e057e-ccc7-485d-9385-9ab94944a701 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:59:14.154146 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-394e057e-ccc7-485d-9385-9ab94944a701 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:59:14.154146 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-394e057e-ccc7-485d-9385-9ab94944a701 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:59:14.173844 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-394e057e-ccc7-485d-9385-9ab94944a701 None None] MFA Rules not processed for user `adcbf206828d49b9bf2ae720f71068bc`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:59:14.175032 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-394e057e-ccc7-485d-9385-9ab94944a701 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:59:14.227660 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1237/2440] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:59:14 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 75 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:59:20.005176 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=32 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:59:24.221271 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=75 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:59:39.281243 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e4515a6-8664-41f4-8e23-bbbf7dc82c4e None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:59:39.281876 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e4515a6-8664-41f4-8e23-bbbf7dc82c4e None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:59:39.281876 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e4515a6-8664-41f4-8e23-bbbf7dc82c4e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:59:39.301618 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-6e4515a6-8664-41f4-8e23-bbbf7dc82c4e None None] MFA Rules not processed for user `5f3247ddcda54394995d844c776bf43c`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:59:39.302852 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-6e4515a6-8664-41f4-8e23-bbbf7dc82c4e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:59:39.357215 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1204/2441] 10.222.0.33 () {58 vars in 1009 bytes} [Sun Oct 9 00:59:39 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:59:39.400232 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-394e057e-ccc7-485d-9385-9ab94944a701 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:59:39.401652 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-394e057e-ccc7-485d-9385-9ab94944a701 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:59:39.478075 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-394e057e-ccc7-485d-9385-9ab94944a701 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:59:39.483107 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ac72964-5480-4b03-8a8c-dd049ca33736 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '70e367e802894b08afd7d756d8d9ecff', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:59:39.483776 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac72964-5480-4b03-8a8c-dd049ca33736 service swift] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:59:39.484062 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac72964-5480-4b03-8a8c-dd049ca33736 service swift] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:59:39.484335 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac72964-5480-4b03-8a8c-dd049ca33736 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:59:39.493083 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ac72964-5480-4b03-8a8c-dd049ca33736 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:59:39.494674 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ac72964-5480-4b03-8a8c-dd049ca33736 service swift] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:59:39.503486 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1238/2442] 10.222.0.33 () {60 vars in 1384 bytes} [Sun Oct 9 00:59:39 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 09 00:59:47.049781 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cb7353e-60c4-43f8-8c52-14a5d1afe14a None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:59:47.050379 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cb7353e-60c4-43f8-8c52-14a5d1afe14a None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:59:47.050379 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cb7353e-60c4-43f8-8c52-14a5d1afe14a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:59:47.070604 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-4cb7353e-60c4-43f8-8c52-14a5d1afe14a None None] MFA Rules not processed for user `b027fb8adadc488e975c4541e7ee6626`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:59:47.072794 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-4cb7353e-60c4-43f8-8c52-14a5d1afe14a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:59:47.159749 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1205/2443] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:59:47 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 09 00:59:49.499567 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=32 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:59:55.977274 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63f14425-de62-4d80-91a6-b7e8973cd3bc None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:59:55.977923 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63f14425-de62-4d80-91a6-b7e8973cd3bc None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:59:55.977923 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63f14425-de62-4d80-91a6-b7e8973cd3bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:59:55.997134 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-63f14425-de62-4d80-91a6-b7e8973cd3bc None None] MFA Rules not processed for user `5f3247ddcda54394995d844c776bf43c`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:59:55.998472 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-63f14425-de62-4d80-91a6-b7e8973cd3bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:59:56.006215 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1239/2444] 10.222.0.33 () {58 vars in 967 bytes} [Sun Oct 9 00:59:55 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:59:56.018454 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cb7353e-60c4-43f8-8c52-14a5d1afe14a None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:59:56.030305 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cb7353e-60c4-43f8-8c52-14a5d1afe14a None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:59:56.035277 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-325a8551-d7aa-4233-97dd-57d1b8ac3c1a service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '70e367e802894b08afd7d756d8d9ecff', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:59:56.035967 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-325a8551-d7aa-4233-97dd-57d1b8ac3c1a service swift] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:59:56.036248 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-325a8551-d7aa-4233-97dd-57d1b8ac3c1a service swift] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:59:56.036516 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-325a8551-d7aa-4233-97dd-57d1b8ac3c1a service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:59:56.044303 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-325a8551-d7aa-4233-97dd-57d1b8ac3c1a service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:59:56.045875 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-325a8551-d7aa-4233-97dd-57d1b8ac3c1a service swift] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:59:56.054249 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1206/2445] 10.222.0.33 () {60 vars in 1384 bytes} [Sun Oct 9 00:59:56 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Oct 09 00:59:58.224055 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63f14425-de62-4d80-91a6-b7e8973cd3bc None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:59:58.225608 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-63f14425-de62-4d80-91a6-b7e8973cd3bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:59:58.292648 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63f14425-de62-4d80-91a6-b7e8973cd3bc None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:59:58.297288 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09b476a3-f186-454d-a060-fd6d67e5b617 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:59:58.297958 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b476a3-f186-454d-a060-fd6d67e5b617 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:59:58.298239 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b476a3-f186-454d-a060-fd6d67e5b617 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:59:58.298549 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b476a3-f186-454d-a060-fd6d67e5b617 admin admin] PATH_INFO: `/v3/users/adcbf206828d49b9bf2ae720f71068bc` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:59:58.299715 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09b476a3-f186-454d-a060-fd6d67e5b617 admin admin] RBAC: Authorizing `identity:delete_user(user_id=adcbf206828d49b9bf2ae720f71068bc)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:59:58.301605 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09b476a3-f186-454d-a060-fd6d67e5b617 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:59:58.348424 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-09b476a3-f186-454d-a060-fd6d67e5b617 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'adcbf206828d49b9bf2ae720f71068bc'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:59:58.355984 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-09b476a3-f186-454d-a060-fd6d67e5b617 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'adcbf206828d49b9bf2ae720f71068bc'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:59:58.356695 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-09b476a3-f186-454d-a060-fd6d67e5b617 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'adcbf206828d49b9bf2ae720f71068bc'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:59:58.357288 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-09b476a3-f186-454d-a060-fd6d67e5b617 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'adcbf206828d49b9bf2ae720f71068bc'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:59:58.364982 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-09b476a3-f186-454d-a060-fd6d67e5b617 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'adcbf206828d49b9bf2ae720f71068bc'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:59:58.383237 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1240/2446] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 00:59:58 2022] DELETE /identity/v3/users/adcbf206828d49b9bf2ae720f71068bc => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:59:58.556650 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-325a8551-d7aa-4233-97dd-57d1b8ac3c1a service swift] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:59:58.568753 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-325a8551-d7aa-4233-97dd-57d1b8ac3c1a service swift] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:59:58.634834 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4213306-28e4-4c93-b8ac-f079f5e54046 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:59:58.635472 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4213306-28e4-4c93-b8ac-f079f5e54046 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:59:58.635778 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4213306-28e4-4c93-b8ac-f079f5e54046 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:59:58.636056 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4213306-28e4-4c93-b8ac-f079f5e54046 admin admin] PATH_INFO: `/v3/projects/5edae5583d2e4a4688bef23f3a14eda4` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:59:58.637169 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4213306-28e4-4c93-b8ac-f079f5e54046 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5edae5583d2e4a4688bef23f3a14eda4)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:59:58.639066 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4213306-28e4-4c93-b8ac-f079f5e54046 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:59:58.683194 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:59:58.693201 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-a4213306-28e4-4c93-b8ac-f079f5e54046 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5edae5583d2e4a4688bef23f3a14eda4', 'request_id': 'req-a4213306-28e4-4c93-b8ac-f079f5e54046'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:59:58.700200 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-a4213306-28e4-4c93-b8ac-f079f5e54046 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5edae5583d2e4a4688bef23f3a14eda4', 'request_id': 'req-a4213306-28e4-4c93-b8ac-f079f5e54046'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:59:58.707124 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-a4213306-28e4-4c93-b8ac-f079f5e54046 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5edae5583d2e4a4688bef23f3a14eda4', 'request_id': 'req-a4213306-28e4-4c93-b8ac-f079f5e54046'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:59:58.721129 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-a4213306-28e4-4c93-b8ac-f079f5e54046 admin admin] The token cache is being invalidate because project 5edae5583d2e4a4688bef23f3a14eda4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81453) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 00:59:58.721424 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-a4213306-28e4-4c93-b8ac-f079f5e54046 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:59:58.722063 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-a4213306-28e4-4c93-b8ac-f079f5e54046 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 00:59:58.723580 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1207/2447] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 00:59:58 2022] DELETE /identity/v3/projects/5edae5583d2e4a4688bef23f3a14eda4 => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:59:58.728588 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c853ec-7f48-4109-8924-2f2e50ea64d4 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:59:58.728951 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c853ec-7f48-4109-8924-2f2e50ea64d4 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:59:58.729333 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c853ec-7f48-4109-8924-2f2e50ea64d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:59:58.749378 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-19c853ec-7f48-4109-8924-2f2e50ea64d4 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:59:58.750579 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-19c853ec-7f48-4109-8924-2f2e50ea64d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:59:58.811902 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1241/2448] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:59:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:59:58.819953 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92544b4-b76f-473c-af4a-2336b1631774 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:59:58.820243 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92544b4-b76f-473c-af4a-2336b1631774 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:59:58.820522 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92544b4-b76f-473c-af4a-2336b1631774 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:59:58.841064 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-c92544b4-b76f-473c-af4a-2336b1631774 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:59:58.842299 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-c92544b4-b76f-473c-af4a-2336b1631774 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:59:58.851057 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1208/2449] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:59:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:59:58.855294 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19c853ec-7f48-4109-8924-2f2e50ea64d4 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:59:58.865681 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19c853ec-7f48-4109-8924-2f2e50ea64d4 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:59:58.870573 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40e664a8-9008-4d28-aaf4-ec8c6cace9c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:59:58.871184 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e664a8-9008-4d28-aaf4-ec8c6cace9c5 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:59:58.871465 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e664a8-9008-4d28-aaf4-ec8c6cace9c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:59:58.871763 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e664a8-9008-4d28-aaf4-ec8c6cace9c5 admin admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:59:58.872536 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40e664a8-9008-4d28-aaf4-ec8c6cace9c5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 00:59:58.872843 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40e664a8-9008-4d28-aaf4-ec8c6cace9c5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:59:58.874472 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40e664a8-9008-4d28-aaf4-ec8c6cace9c5 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:59:58.885379 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 00:59:58.888973 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1242/2450] 10.222.0.33 () {58 vars in 1153 bytes} [Sun Oct 9 00:59:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 00:59:58.893016 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c92544b4-b76f-473c-af4a-2336b1631774 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:59:58.902915 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c92544b4-b76f-473c-af4a-2336b1631774 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:59:58.907544 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e78d4087-32c3-4ba9-8854-d65304ae7676 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:59:58.908171 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e78d4087-32c3-4ba9-8854-d65304ae7676 admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:59:58.908451 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e78d4087-32c3-4ba9-8854-d65304ae7676 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:59:58.908720 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e78d4087-32c3-4ba9-8854-d65304ae7676 admin admin] PATH_INFO: `/v3/projects` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:59:58.909348 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e78d4087-32c3-4ba9-8854-d65304ae7676 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:59:58.911267 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e78d4087-32c3-4ba9-8854-d65304ae7676 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:59:58.938791 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1209/2451] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 00:59:58 2022] POST /identity/v3/projects => generated 385 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:59:58.943438 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40e664a8-9008-4d28-aaf4-ec8c6cace9c5 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:59:58.953418 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40e664a8-9008-4d28-aaf4-ec8c6cace9c5 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:59:59.010913 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c44786ca-3b9c-4ba3-9d3e-a32af5ef14d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:59:59.011518 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c44786ca-3b9c-4ba3-9d3e-a32af5ef14d0 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:59:59.011849 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c44786ca-3b9c-4ba3-9d3e-a32af5ef14d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:59:59.012128 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c44786ca-3b9c-4ba3-9d3e-a32af5ef14d0 admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:59:59.012752 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c44786ca-3b9c-4ba3-9d3e-a32af5ef14d0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:59:59.014658 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c44786ca-3b9c-4ba3-9d3e-a32af5ef14d0 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:59:59.039405 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1243/2452] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 00:59:58 2022] POST /identity/v3/users => generated 293 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 00:59:59.041076 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e78d4087-32c3-4ba9-8854-d65304ae7676 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:59:59.051831 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e78d4087-32c3-4ba9-8854-d65304ae7676 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:59:59.056425 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a88b63a-5bf1-4d44-98f4-5bbd73f4c8ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:59:59.057015 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a88b63a-5bf1-4d44-98f4-5bbd73f4c8ea admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:59:59.057293 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a88b63a-5bf1-4d44-98f4-5bbd73f4c8ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:59:59.057566 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a88b63a-5bf1-4d44-98f4-5bbd73f4c8ea admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:59:59.058196 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a88b63a-5bf1-4d44-98f4-5bbd73f4c8ea admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:59:59.059991 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a88b63a-5bf1-4d44-98f4-5bbd73f4c8ea admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:59:59.073988 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1210/2453] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:59:59 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:59:59.075648 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c44786ca-3b9c-4ba3-9d3e-a32af5ef14d0 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:59:59.090840 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c44786ca-3b9c-4ba3-9d3e-a32af5ef14d0 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:59:59.096422 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17f1e4bb-f73d-4e9c-aeba-eb52c198cc1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:59:59.097634 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17f1e4bb-f73d-4e9c-aeba-eb52c198cc1d admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:59:59.098041 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17f1e4bb-f73d-4e9c-aeba-eb52c198cc1d admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:59:59.098388 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17f1e4bb-f73d-4e9c-aeba-eb52c198cc1d admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:59:59.099272 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17f1e4bb-f73d-4e9c-aeba-eb52c198cc1d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:59:59.100931 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17f1e4bb-f73d-4e9c-aeba-eb52c198cc1d admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:59:59.112209 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1244/2454] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 00:59:59 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 00:59:59.117136 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a88b63a-5bf1-4d44-98f4-5bbd73f4c8ea admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:59:59.128363 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a88b63a-5bf1-4d44-98f4-5bbd73f4c8ea admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:59:59.134241 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24ec0f0d-d011-4735-85b3-fac3cfd1cde8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:59:59.134891 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24ec0f0d-d011-4735-85b3-fac3cfd1cde8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:59:59.135168 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24ec0f0d-d011-4735-85b3-fac3cfd1cde8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:59:59.135438 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24ec0f0d-d011-4735-85b3-fac3cfd1cde8 admin admin] PATH_INFO: `/v3/projects/f766c4270e944753b745acc0b6bce492/users/ed713d25fa2a4ecab59e25e7ede9f93e/roles/588016933bab436d8b7c3ac76827356d` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:59:59.150863 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24ec0f0d-d011-4735-85b3-fac3cfd1cde8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f766c4270e944753b745acc0b6bce492, user_id=ed713d25fa2a4ecab59e25e7ede9f93e, role_id=588016933bab436d8b7c3ac76827356d)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:59:59.153844 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24ec0f0d-d011-4735-85b3-fac3cfd1cde8 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:59:59.167535 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1211/2455] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 00:59:59 2022] PUT /identity/v3/projects/f766c4270e944753b745acc0b6bce492/users/ed713d25fa2a4ecab59e25e7ede9f93e/roles/588016933bab436d8b7c3ac76827356d => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 00:59:59.179431 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf23f6be-d7d5-4d6e-a6ff-164de0da9d5b None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:59:59.179776 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf23f6be-d7d5-4d6e-a6ff-164de0da9d5b None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:59:59.180225 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf23f6be-d7d5-4d6e-a6ff-164de0da9d5b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:59:59.208304 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-bf23f6be-d7d5-4d6e-a6ff-164de0da9d5b None None] MFA Rules not processed for user `ed713d25fa2a4ecab59e25e7ede9f93e`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 00:59:59.209512 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-bf23f6be-d7d5-4d6e-a6ff-164de0da9d5b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:59:59.276621 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1245/2456] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 00:59:59 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 00:59:59.288439 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24ec0f0d-d011-4735-85b3-fac3cfd1cde8 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 00:59:59.290227 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-24ec0f0d-d011-4735-85b3-fac3cfd1cde8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 00:59:59.346634 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24ec0f0d-d011-4735-85b3-fac3cfd1cde8 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 00:59:59.351021 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1011168b-db16-4e23-ba01-8883b0ef19bd service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3f48335668a4e8f934ac11ff66cd308', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 00:59:59.351662 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1011168b-db16-4e23-ba01-8883b0ef19bd service cinder] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 00:59:59.351947 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1011168b-db16-4e23-ba01-8883b0ef19bd service cinder] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 00:59:59.352224 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1011168b-db16-4e23-ba01-8883b0ef19bd service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 00:59:59.361214 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1011168b-db16-4e23-ba01-8883b0ef19bd service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 00:59:59.363181 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1011168b-db16-4e23-ba01-8883b0ef19bd service cinder] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 00:59:59.372178 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1212/2457] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 00:59:59 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 01:00:01.928612 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf23f6be-d7d5-4d6e-a6ff-164de0da9d5b None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:00:01.989711 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf23f6be-d7d5-4d6e-a6ff-164de0da9d5b None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:00:01.994176 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05442adb-5e3a-4fa1-a07d-d46a28c9b661 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:00:01.994794 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05442adb-5e3a-4fa1-a07d-d46a28c9b661 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:00:01.995073 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05442adb-5e3a-4fa1-a07d-d46a28c9b661 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:00:01.995347 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05442adb-5e3a-4fa1-a07d-d46a28c9b661 admin admin] PATH_INFO: `/v3/users/ed713d25fa2a4ecab59e25e7ede9f93e` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:00:01.996481 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05442adb-5e3a-4fa1-a07d-d46a28c9b661 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ed713d25fa2a4ecab59e25e7ede9f93e)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:00:01.998382 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05442adb-5e3a-4fa1-a07d-d46a28c9b661 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:00:02.009478 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:00:02.044458 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-05442adb-5e3a-4fa1-a07d-d46a28c9b661 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ed713d25fa2a4ecab59e25e7ede9f93e'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:00:02.052115 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-05442adb-5e3a-4fa1-a07d-d46a28c9b661 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ed713d25fa2a4ecab59e25e7ede9f93e'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:00:02.052783 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-05442adb-5e3a-4fa1-a07d-d46a28c9b661 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ed713d25fa2a4ecab59e25e7ede9f93e'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:00:02.053380 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-05442adb-5e3a-4fa1-a07d-d46a28c9b661 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ed713d25fa2a4ecab59e25e7ede9f93e'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:00:02.060504 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-05442adb-5e3a-4fa1-a07d-d46a28c9b661 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ed713d25fa2a4ecab59e25e7ede9f93e'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:00:02.079631 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1246/2458] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 01:00:01 2022] DELETE /identity/v3/users/ed713d25fa2a4ecab59e25e7ede9f93e => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:00:02.082884 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3897f35-56d1-4be9-b31b-d82997eec5a7 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:00:02.083326 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3897f35-56d1-4be9-b31b-d82997eec5a7 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:00:02.083931 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3897f35-56d1-4be9-b31b-d82997eec5a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:00:02.102452 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-c3897f35-56d1-4be9-b31b-d82997eec5a7 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:00:02.103707 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-c3897f35-56d1-4be9-b31b-d82997eec5a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:00:02.162191 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1213/2459] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:00:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:00:02.172016 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05442adb-5e3a-4fa1-a07d-d46a28c9b661 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:00:02.174157 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-05442adb-5e3a-4fa1-a07d-d46a28c9b661 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:00:02.235035 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05442adb-5e3a-4fa1-a07d-d46a28c9b661 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:00:02.239163 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0041de84-83a1-4699-84e9-de5eef7c8f3e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:00:02.239862 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0041de84-83a1-4699-84e9-de5eef7c8f3e service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:00:02.240071 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0041de84-83a1-4699-84e9-de5eef7c8f3e service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:00:02.240343 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0041de84-83a1-4699-84e9-de5eef7c8f3e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:00:02.249577 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0041de84-83a1-4699-84e9-de5eef7c8f3e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:00:02.250991 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0041de84-83a1-4699-84e9-de5eef7c8f3e service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:00:02.261222 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1247/2460] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 01:00:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 01:00:02.634348 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3897f35-56d1-4be9-b31b-d82997eec5a7 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:00:02.635816 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-c3897f35-56d1-4be9-b31b-d82997eec5a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:00:02.648354 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3897f35-56d1-4be9-b31b-d82997eec5a7 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:00:02.653298 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29e3bc88-bfbb-4492-b0fd-99170800e181 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:00:02.654055 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e3bc88-bfbb-4492-b0fd-99170800e181 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:00:02.654259 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e3bc88-bfbb-4492-b0fd-99170800e181 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:00:02.654469 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e3bc88-bfbb-4492-b0fd-99170800e181 admin admin] PATH_INFO: `/v3/projects/f766c4270e944753b745acc0b6bce492` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:00:02.655621 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29e3bc88-bfbb-4492-b0fd-99170800e181 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f766c4270e944753b745acc0b6bce492)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:00:02.657543 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29e3bc88-bfbb-4492-b0fd-99170800e181 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:00:02.659263 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=93,keystone:DELETE=4,keystone:INSERT=3 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:00:02.712940 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-29e3bc88-bfbb-4492-b0fd-99170800e181 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f766c4270e944753b745acc0b6bce492', 'request_id': 'req-29e3bc88-bfbb-4492-b0fd-99170800e181'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:00:02.718413 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-29e3bc88-bfbb-4492-b0fd-99170800e181 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f766c4270e944753b745acc0b6bce492', 'request_id': 'req-29e3bc88-bfbb-4492-b0fd-99170800e181'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:00:02.725386 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-29e3bc88-bfbb-4492-b0fd-99170800e181 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f766c4270e944753b745acc0b6bce492', 'request_id': 'req-29e3bc88-bfbb-4492-b0fd-99170800e181'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:00:02.738082 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-29e3bc88-bfbb-4492-b0fd-99170800e181 admin admin] The token cache is being invalidate because project f766c4270e944753b745acc0b6bce492 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81453) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 01:00:02.738370 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-29e3bc88-bfbb-4492-b0fd-99170800e181 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:00:02.738996 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-29e3bc88-bfbb-4492-b0fd-99170800e181 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:00:02.740707 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1214/2461] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 01:00:02 2022] DELETE /identity/v3/projects/f766c4270e944753b745acc0b6bce492 => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:00:02.746326 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed220536-96f8-4754-bb80-fa6828ee7877 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:00:02.746677 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed220536-96f8-4754-bb80-fa6828ee7877 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:00:02.746956 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed220536-96f8-4754-bb80-fa6828ee7877 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:00:02.767064 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-ed220536-96f8-4754-bb80-fa6828ee7877 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:00:02.768294 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-ed220536-96f8-4754-bb80-fa6828ee7877 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:00:02.828171 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1248/2462] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:00:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:00:02.836682 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46de11cf-f116-4b2a-b1d1-3ba66803f123 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:00:02.836968 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46de11cf-f116-4b2a-b1d1-3ba66803f123 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:00:02.837276 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46de11cf-f116-4b2a-b1d1-3ba66803f123 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:00:02.856755 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-46de11cf-f116-4b2a-b1d1-3ba66803f123 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:00:02.857974 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-46de11cf-f116-4b2a-b1d1-3ba66803f123 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:00:02.866700 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1215/2463] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:00:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:00:02.871774 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed220536-96f8-4754-bb80-fa6828ee7877 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:00:02.882779 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed220536-96f8-4754-bb80-fa6828ee7877 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:00:02.887486 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10e6f11a-2668-4c2e-93ea-32f99121ae0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:00:02.888143 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10e6f11a-2668-4c2e-93ea-32f99121ae0f admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:00:02.888419 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10e6f11a-2668-4c2e-93ea-32f99121ae0f admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:00:02.888688 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10e6f11a-2668-4c2e-93ea-32f99121ae0f admin admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:00:02.889319 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10e6f11a-2668-4c2e-93ea-32f99121ae0f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 01:00:02.889617 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10e6f11a-2668-4c2e-93ea-32f99121ae0f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:00:02.891192 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10e6f11a-2668-4c2e-93ea-32f99121ae0f admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:00:02.903875 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1249/2464] 10.222.0.33 () {58 vars in 1153 bytes} [Sun Oct 9 01:00:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 01:00:02.908154 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46de11cf-f116-4b2a-b1d1-3ba66803f123 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:00:02.917313 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46de11cf-f116-4b2a-b1d1-3ba66803f123 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:00:02.921945 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60c8043c-7ce4-41e4-ba4e-547afb07b6a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:00:02.922536 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c8043c-7ce4-41e4-ba4e-547afb07b6a3 admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:00:02.922819 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c8043c-7ce4-41e4-ba4e-547afb07b6a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:00:02.923085 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c8043c-7ce4-41e4-ba4e-547afb07b6a3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:00:02.923740 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60c8043c-7ce4-41e4-ba4e-547afb07b6a3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:00:02.925646 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60c8043c-7ce4-41e4-ba4e-547afb07b6a3 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:00:02.951998 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1216/2465] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 01:00:02 2022] POST /identity/v3/projects => generated 375 bytes in 44 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 09 01:00:02.956760 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10e6f11a-2668-4c2e-93ea-32f99121ae0f admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:00:02.967141 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10e6f11a-2668-4c2e-93ea-32f99121ae0f admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:00:02.979841 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:00:03.026462 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2453c1d7-34f7-4689-9a21-667cd062c2f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:00:03.026992 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2453c1d7-34f7-4689-9a21-667cd062c2f4 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:00:03.027276 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2453c1d7-34f7-4689-9a21-667cd062c2f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:00:03.027848 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2453c1d7-34f7-4689-9a21-667cd062c2f4 admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:00:03.028509 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2453c1d7-34f7-4689-9a21-667cd062c2f4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:00:03.030451 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2453c1d7-34f7-4689-9a21-667cd062c2f4 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:00:03.051980 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1250/2466] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 01:00:02 2022] POST /identity/v3/users => generated 288 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 01:00:03.055719 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60c8043c-7ce4-41e4-ba4e-547afb07b6a3 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:00:03.066339 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60c8043c-7ce4-41e4-ba4e-547afb07b6a3 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:00:03.071098 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd2efe27-cf58-4aa5-ba10-e7ebbfc4e8a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:00:03.071739 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd2efe27-cf58-4aa5-ba10-e7ebbfc4e8a5 admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:00:03.072016 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd2efe27-cf58-4aa5-ba10-e7ebbfc4e8a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:00:03.072291 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd2efe27-cf58-4aa5-ba10-e7ebbfc4e8a5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:00:03.072921 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd2efe27-cf58-4aa5-ba10-e7ebbfc4e8a5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:00:03.074535 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd2efe27-cf58-4aa5-ba10-e7ebbfc4e8a5 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:00:03.085640 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1217/2467] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 01:00:03 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 09 01:00:03.089452 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2453c1d7-34f7-4689-9a21-667cd062c2f4 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:00:03.099821 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2453c1d7-34f7-4689-9a21-667cd062c2f4 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:00:03.104371 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5908960-a942-4cfd-a9d5-0042b9cc08c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:00:03.104965 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5908960-a942-4cfd-a9d5-0042b9cc08c7 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:00:03.105244 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5908960-a942-4cfd-a9d5-0042b9cc08c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:00:03.105522 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5908960-a942-4cfd-a9d5-0042b9cc08c7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:00:03.106156 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5908960-a942-4cfd-a9d5-0042b9cc08c7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:00:03.107773 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5908960-a942-4cfd-a9d5-0042b9cc08c7 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:00:03.118331 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1251/2468] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 01:00:03 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 01:00:03.122399 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd2efe27-cf58-4aa5-ba10-e7ebbfc4e8a5 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:00:03.132892 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd2efe27-cf58-4aa5-ba10-e7ebbfc4e8a5 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:00:03.137471 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9bc8a38-8c48-465a-8728-939cdd499ab4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:00:03.138117 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9bc8a38-8c48-465a-8728-939cdd499ab4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:00:03.138395 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9bc8a38-8c48-465a-8728-939cdd499ab4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:00:03.138669 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9bc8a38-8c48-465a-8728-939cdd499ab4 admin admin] PATH_INFO: `/v3/projects/9a6e25abae6b4b499a402bc4585bd06f/users/5becb8eb7b8d4c619f8164de3389a1ce/roles/588016933bab436d8b7c3ac76827356d` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:00:03.153755 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9bc8a38-8c48-465a-8728-939cdd499ab4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9a6e25abae6b4b499a402bc4585bd06f, user_id=5becb8eb7b8d4c619f8164de3389a1ce, role_id=588016933bab436d8b7c3ac76827356d)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:00:03.156445 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9bc8a38-8c48-465a-8728-939cdd499ab4 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:00:03.169751 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1218/2469] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 01:00:03 2022] PUT /identity/v3/projects/9a6e25abae6b4b499a402bc4585bd06f/users/5becb8eb7b8d4c619f8164de3389a1ce/roles/588016933bab436d8b7c3ac76827356d => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:00:03.179964 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3645cd8b-3af9-4e22-95f9-28cbe71b1830 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:00:03.180258 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3645cd8b-3af9-4e22-95f9-28cbe71b1830 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:00:03.180533 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3645cd8b-3af9-4e22-95f9-28cbe71b1830 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:00:03.209952 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-3645cd8b-3af9-4e22-95f9-28cbe71b1830 None None] MFA Rules not processed for user `5becb8eb7b8d4c619f8164de3389a1ce`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:00:03.211155 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-3645cd8b-3af9-4e22-95f9-28cbe71b1830 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:00:03.274518 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1252/2470] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:00:03 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:00:03.285841 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9bc8a38-8c48-465a-8728-939cdd499ab4 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:00:03.287509 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-c9bc8a38-8c48-465a-8728-939cdd499ab4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:00:03.344692 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9bc8a38-8c48-465a-8728-939cdd499ab4 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:00:03.348410 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38a84c1f-5386-4549-aad3-68e08d713ebb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3f48335668a4e8f934ac11ff66cd308', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:00:03.349009 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38a84c1f-5386-4549-aad3-68e08d713ebb service cinder] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:00:03.349286 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38a84c1f-5386-4549-aad3-68e08d713ebb service cinder] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:00:03.349554 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38a84c1f-5386-4549-aad3-68e08d713ebb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:00:03.357688 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38a84c1f-5386-4549-aad3-68e08d713ebb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:00:03.359098 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38a84c1f-5386-4549-aad3-68e08d713ebb service cinder] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:00:03.368251 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1219/2471] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 01:00:03 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 09 01:00:04.729244 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d3fe35-8adc-4519-995c-674c8a7c8bbe None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:00:04.729858 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d3fe35-8adc-4519-995c-674c8a7c8bbe None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:00:04.729858 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d3fe35-8adc-4519-995c-674c8a7c8bbe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:00:04.748983 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-d6d3fe35-8adc-4519-995c-674c8a7c8bbe None None] MFA Rules not processed for user `5becb8eb7b8d4c619f8164de3389a1ce`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:00:04.750229 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-d6d3fe35-8adc-4519-995c-674c8a7c8bbe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:00:04.757848 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1253/2472] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:00:04 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:00:13.364654 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=77,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:00:14.743375 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=61,keystone:INSERT=3 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:00:19.809737 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82343941-df69-4900-9660-1ef5f8ab67d5 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:00:19.810192 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82343941-df69-4900-9660-1ef5f8ab67d5 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:00:19.810285 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82343941-df69-4900-9660-1ef5f8ab67d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:00:19.828966 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-82343941-df69-4900-9660-1ef5f8ab67d5 None None] MFA Rules not processed for user `5f3247ddcda54394995d844c776bf43c`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:00:19.830142 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-82343941-df69-4900-9660-1ef5f8ab67d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:00:19.887452 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1220/2473] 10.222.0.33 () {58 vars in 1009 bytes} [Sun Oct 9 01:00:19 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:00:19.900539 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6d3fe35-8adc-4519-995c-674c8a7c8bbe None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:00:19.902266 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-d6d3fe35-8adc-4519-995c-674c8a7c8bbe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:00:19.974558 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6d3fe35-8adc-4519-995c-674c8a7c8bbe None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:00:19.979489 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f359939f-489c-411f-86d0-f6f6477e4337 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '70e367e802894b08afd7d756d8d9ecff', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:00:19.980164 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f359939f-489c-411f-86d0-f6f6477e4337 service swift] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:00:19.980439 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f359939f-489c-411f-86d0-f6f6477e4337 service swift] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:00:19.980715 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f359939f-489c-411f-86d0-f6f6477e4337 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:00:19.989353 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f359939f-489c-411f-86d0-f6f6477e4337 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:00:19.990925 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f359939f-489c-411f-86d0-f6f6477e4337 service swift] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:00:19.999236 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1254/2474] 10.222.0.33 () {60 vars in 1384 bytes} [Sun Oct 9 01:00:19 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 09 01:00:20.503456 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ca594a-8134-4b2a-9ae1-3952ace8f807 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:00:20.503764 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ca594a-8134-4b2a-9ae1-3952ace8f807 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:00:20.504598 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ca594a-8134-4b2a-9ae1-3952ace8f807 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:00:20.523257 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-77ca594a-8134-4b2a-9ae1-3952ace8f807 None None] MFA Rules not processed for user `5f3247ddcda54394995d844c776bf43c`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:00:20.524606 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-77ca594a-8134-4b2a-9ae1-3952ace8f807 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:00:20.531899 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1221/2475] 10.222.0.33 () {58 vars in 967 bytes} [Sun Oct 9 01:00:20 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:00:20.543869 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f359939f-489c-411f-86d0-f6f6477e4337 service swift] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:00:20.554835 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f359939f-489c-411f-86d0-f6f6477e4337 service swift] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:00:20.559825 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d658a9b-f6d6-4ee7-81bf-7ec260d072f7 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '70e367e802894b08afd7d756d8d9ecff', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:00:20.560411 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d658a9b-f6d6-4ee7-81bf-7ec260d072f7 service swift] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:00:20.560691 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d658a9b-f6d6-4ee7-81bf-7ec260d072f7 service swift] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:00:20.560964 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d658a9b-f6d6-4ee7-81bf-7ec260d072f7 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:00:20.569440 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d658a9b-f6d6-4ee7-81bf-7ec260d072f7 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:00:20.571037 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d658a9b-f6d6-4ee7-81bf-7ec260d072f7 service swift] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:00:20.579117 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1255/2476] 10.222.0.33 () {60 vars in 1384 bytes} [Sun Oct 9 01:00:20 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 09 01:00:20.802301 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7088836-178e-4b79-967d-90c8da7fd7d4 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:00:20.802584 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7088836-178e-4b79-967d-90c8da7fd7d4 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:00:20.802854 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7088836-178e-4b79-967d-90c8da7fd7d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:00:20.821507 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-f7088836-178e-4b79-967d-90c8da7fd7d4 None None] MFA Rules not processed for user `5f3247ddcda54394995d844c776bf43c`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:00:20.822700 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-f7088836-178e-4b79-967d-90c8da7fd7d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:00:20.829676 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1222/2477] 10.222.0.33 () {58 vars in 1009 bytes} [Sun Oct 9 01:00:20 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:00:20.841419 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d658a9b-f6d6-4ee7-81bf-7ec260d072f7 service swift] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:00:20.852215 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d658a9b-f6d6-4ee7-81bf-7ec260d072f7 service swift] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:00:20.857178 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9146cf4d-c1a4-4522-830d-dcc0dbdfa125 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '70e367e802894b08afd7d756d8d9ecff', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:00:20.857757 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9146cf4d-c1a4-4522-830d-dcc0dbdfa125 service swift] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:00:20.858037 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9146cf4d-c1a4-4522-830d-dcc0dbdfa125 service swift] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:00:20.858312 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9146cf4d-c1a4-4522-830d-dcc0dbdfa125 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:00:20.866750 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9146cf4d-c1a4-4522-830d-dcc0dbdfa125 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:00:20.868352 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9146cf4d-c1a4-4522-830d-dcc0dbdfa125 service swift] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:00:20.876918 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1256/2478] 10.222.0.33 () {60 vars in 1384 bytes} [Sun Oct 9 01:00:20 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Oct 09 01:00:21.210883 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b95e6530-fa68-4333-89e3-555c4c7828cf None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:00:21.211174 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b95e6530-fa68-4333-89e3-555c4c7828cf None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:00:21.211450 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b95e6530-fa68-4333-89e3-555c4c7828cf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:00:21.230887 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-b95e6530-fa68-4333-89e3-555c4c7828cf None None] MFA Rules not processed for user `5f3247ddcda54394995d844c776bf43c`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:00:21.232153 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-b95e6530-fa68-4333-89e3-555c4c7828cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:00:21.239287 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1223/2479] 10.222.0.33 () {58 vars in 967 bytes} [Sun Oct 9 01:00:21 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:00:21.251377 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9146cf4d-c1a4-4522-830d-dcc0dbdfa125 service swift] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:00:21.261762 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9146cf4d-c1a4-4522-830d-dcc0dbdfa125 service swift] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:00:21.266715 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68ae9d86-ab43-4d6e-b15a-de6c73f0a2ef service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '70e367e802894b08afd7d756d8d9ecff', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:00:21.267298 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ae9d86-ab43-4d6e-b15a-de6c73f0a2ef service swift] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:00:21.267576 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ae9d86-ab43-4d6e-b15a-de6c73f0a2ef service swift] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:00:21.267879 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ae9d86-ab43-4d6e-b15a-de6c73f0a2ef service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:00:21.276190 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68ae9d86-ab43-4d6e-b15a-de6c73f0a2ef service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:00:21.278206 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68ae9d86-ab43-4d6e-b15a-de6c73f0a2ef service swift] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:00:21.286663 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1257/2480] 10.222.0.33 () {60 vars in 1384 bytes} [Sun Oct 9 01:00:21 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 09 01:00:23.365234 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b95e6530-fa68-4333-89e3-555c4c7828cf None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:00:23.428066 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b95e6530-fa68-4333-89e3-555c4c7828cf None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:00:23.433609 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bbf8f84-96c9-4843-b5d9-4d88dbcaf5ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:00:23.433609 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bbf8f84-96c9-4843-b5d9-4d88dbcaf5ce admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:00:23.433609 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bbf8f84-96c9-4843-b5d9-4d88dbcaf5ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:00:23.433937 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bbf8f84-96c9-4843-b5d9-4d88dbcaf5ce admin admin] PATH_INFO: `/v3/users/5becb8eb7b8d4c619f8164de3389a1ce` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:00:23.434909 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bbf8f84-96c9-4843-b5d9-4d88dbcaf5ce admin admin] RBAC: Authorizing `identity:delete_user(user_id=5becb8eb7b8d4c619f8164de3389a1ce)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:00:23.436837 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bbf8f84-96c9-4843-b5d9-4d88dbcaf5ce admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:00:23.486326 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-4bbf8f84-96c9-4843-b5d9-4d88dbcaf5ce admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5becb8eb7b8d4c619f8164de3389a1ce'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:00:23.494117 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-4bbf8f84-96c9-4843-b5d9-4d88dbcaf5ce admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5becb8eb7b8d4c619f8164de3389a1ce'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:00:23.494797 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-4bbf8f84-96c9-4843-b5d9-4d88dbcaf5ce admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5becb8eb7b8d4c619f8164de3389a1ce'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:00:23.502772 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-4bbf8f84-96c9-4843-b5d9-4d88dbcaf5ce admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5becb8eb7b8d4c619f8164de3389a1ce'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:00:23.515741 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-4bbf8f84-96c9-4843-b5d9-4d88dbcaf5ce admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5becb8eb7b8d4c619f8164de3389a1ce'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:00:23.521827 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1224/2481] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 01:00:23 2022] DELETE /identity/v3/users/5becb8eb7b8d4c619f8164de3389a1ce => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:00:23.526206 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2725998d-6ad6-453f-a2e4-1260529ba976 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:00:23.526692 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2725998d-6ad6-453f-a2e4-1260529ba976 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:00:23.526801 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2725998d-6ad6-453f-a2e4-1260529ba976 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:00:23.545829 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-2725998d-6ad6-453f-a2e4-1260529ba976 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:00:23.547021 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-2725998d-6ad6-453f-a2e4-1260529ba976 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:00:23.611627 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1258/2482] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:00:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 09 01:00:23.620862 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bbf8f84-96c9-4843-b5d9-4d88dbcaf5ce admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:00:23.622414 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-4bbf8f84-96c9-4843-b5d9-4d88dbcaf5ce admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:00:23.629021 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:00:23.683270 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bbf8f84-96c9-4843-b5d9-4d88dbcaf5ce admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:00:23.686926 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69e4ce37-48c7-4a33-bef3-bc6db0c826a3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:00:23.687530 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e4ce37-48c7-4a33-bef3-bc6db0c826a3 service neutron] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:00:23.687853 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e4ce37-48c7-4a33-bef3-bc6db0c826a3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:00:23.688124 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e4ce37-48c7-4a33-bef3-bc6db0c826a3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:00:23.697134 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69e4ce37-48c7-4a33-bef3-bc6db0c826a3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:00:23.698549 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69e4ce37-48c7-4a33-bef3-bc6db0c826a3 service neutron] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:00:23.708894 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1225/2483] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 01:00:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 01:00:23.889533 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2725998d-6ad6-453f-a2e4-1260529ba976 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:00:23.891005 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-2725998d-6ad6-453f-a2e4-1260529ba976 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:00:23.904270 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2725998d-6ad6-453f-a2e4-1260529ba976 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:00:23.908820 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-217488a2-27d6-4b7c-bb0d-b1e693a8796a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:00:23.909444 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-217488a2-27d6-4b7c-bb0d-b1e693a8796a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:00:23.909720 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-217488a2-27d6-4b7c-bb0d-b1e693a8796a admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:00:23.909988 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-217488a2-27d6-4b7c-bb0d-b1e693a8796a admin admin] PATH_INFO: `/v3/projects/9a6e25abae6b4b499a402bc4585bd06f` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:00:23.911753 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-217488a2-27d6-4b7c-bb0d-b1e693a8796a admin admin] RBAC: Authorizing `identity:delete_project(project_id=9a6e25abae6b4b499a402bc4585bd06f)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:00:23.913052 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-217488a2-27d6-4b7c-bb0d-b1e693a8796a admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:00:23.957709 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:00:23.961265 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-217488a2-27d6-4b7c-bb0d-b1e693a8796a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9a6e25abae6b4b499a402bc4585bd06f', 'request_id': 'req-217488a2-27d6-4b7c-bb0d-b1e693a8796a'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:00:23.967282 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-217488a2-27d6-4b7c-bb0d-b1e693a8796a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9a6e25abae6b4b499a402bc4585bd06f', 'request_id': 'req-217488a2-27d6-4b7c-bb0d-b1e693a8796a'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:00:23.973528 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-217488a2-27d6-4b7c-bb0d-b1e693a8796a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9a6e25abae6b4b499a402bc4585bd06f', 'request_id': 'req-217488a2-27d6-4b7c-bb0d-b1e693a8796a'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:00:23.985208 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-217488a2-27d6-4b7c-bb0d-b1e693a8796a admin admin] The token cache is being invalidate because project 9a6e25abae6b4b499a402bc4585bd06f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 01:00:23.985499 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-217488a2-27d6-4b7c-bb0d-b1e693a8796a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:00:23.986125 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-217488a2-27d6-4b7c-bb0d-b1e693a8796a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:00:23.987640 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1259/2484] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 01:00:23 2022] DELETE /identity/v3/projects/9a6e25abae6b4b499a402bc4585bd06f => generated 0 bytes in 99 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:00:28.427693 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7e4fd9-0c39-421f-ae41-12532dad4023 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:00:28.428240 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7e4fd9-0c39-421f-ae41-12532dad4023 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:00:28.428240 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7e4fd9-0c39-421f-ae41-12532dad4023 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:00:28.447288 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-9c7e4fd9-0c39-421f-ae41-12532dad4023 None None] MFA Rules not processed for user `438446d1e6b3402bb18b547c089391f8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:00:28.448541 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-9c7e4fd9-0c39-421f-ae41-12532dad4023 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:00:28.511730 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1226/2485] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:00:28 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:00:30.223932 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68dca29f-118a-4169-bef6-4927d22e0c63 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:00:30.224431 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68dca29f-118a-4169-bef6-4927d22e0c63 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:00:30.224537 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68dca29f-118a-4169-bef6-4927d22e0c63 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:00:30.244357 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-68dca29f-118a-4169-bef6-4927d22e0c63 None None] MFA Rules not processed for user `1512c91201f14b75a9be35b89c2097ef`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:00:30.245549 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-68dca29f-118a-4169-bef6-4927d22e0c63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:00:30.298957 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1260/2486] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:00:30 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 76 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:00:38.503580 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=49 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:00:40.292519 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:DELETE=4,keystone:SELECT=32,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:00:53.214599 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b741c52-8cdb-4ff2-b540-2504141c936f None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:00:53.215135 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b741c52-8cdb-4ff2-b540-2504141c936f None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:00:53.215135 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b741c52-8cdb-4ff2-b540-2504141c936f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:00:53.235106 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-1b741c52-8cdb-4ff2-b540-2504141c936f None None] MFA Rules not processed for user `b027fb8adadc488e975c4541e7ee6626`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:00:53.236342 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-1b741c52-8cdb-4ff2-b540-2504141c936f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:00:53.290217 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1227/2487] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:00:53 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:00:53.327512 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68dca29f-118a-4169-bef6-4927d22e0c63 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:00:53.328931 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-68dca29f-118a-4169-bef6-4927d22e0c63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:00:53.384644 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68dca29f-118a-4169-bef6-4927d22e0c63 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:00:53.388115 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f361658b-8892-4601-b753-93fe655747ad service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1f7a88094d73471787cc0a287adbd394', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:00:53.388709 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f361658b-8892-4601-b753-93fe655747ad service glance] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:00:53.388984 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f361658b-8892-4601-b753-93fe655747ad service glance] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:00:53.389252 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f361658b-8892-4601-b753-93fe655747ad service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:00:53.390914 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-f361658b-8892-4601-b753-93fe655747ad service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:00:53.461364 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f361658b-8892-4601-b753-93fe655747ad service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:00:53.462823 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f361658b-8892-4601-b753-93fe655747ad service glance] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:00:53.474211 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1261/2488] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 01:00:53 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 147 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 01:00:54.053108 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b741c52-8cdb-4ff2-b540-2504141c936f None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:00:54.054487 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-1b741c52-8cdb-4ff2-b540-2504141c936f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:00:54.114441 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b741c52-8cdb-4ff2-b540-2504141c936f None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:00:54.118039 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c37e40b0-3571-43a0-ba9f-457448f90795 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:00:54.118631 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37e40b0-3571-43a0-ba9f-457448f90795 service neutron] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:00:54.118909 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37e40b0-3571-43a0-ba9f-457448f90795 service neutron] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:00:54.119187 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37e40b0-3571-43a0-ba9f-457448f90795 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:00:54.120948 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-c37e40b0-3571-43a0-ba9f-457448f90795 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:00:54.134513 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c37e40b0-3571-43a0-ba9f-457448f90795 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:00:54.135975 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c37e40b0-3571-43a0-ba9f-457448f90795 service neutron] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:00:54.147074 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1228/2489] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 01:00:54 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 01:01:03.468787 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=54 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:01:04.141970 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=51 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:01:09.942866 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f361658b-8892-4601-b753-93fe655747ad service glance] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:01:09.944299 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-f361658b-8892-4601-b753-93fe655747ad service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:01:09.955358 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f361658b-8892-4601-b753-93fe655747ad service glance] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:01:09.960245 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4aa0afe-a255-4ddf-90b4-74ce9b6de32b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:01:09.960837 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4aa0afe-a255-4ddf-90b4-74ce9b6de32b service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:01:09.961115 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4aa0afe-a255-4ddf-90b4-74ce9b6de32b service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:01:09.961387 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4aa0afe-a255-4ddf-90b4-74ce9b6de32b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:01:09.963181 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-c4aa0afe-a255-4ddf-90b4-74ce9b6de32b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:01:09.974316 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4aa0afe-a255-4ddf-90b4-74ce9b6de32b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:01:09.975754 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4aa0afe-a255-4ddf-90b4-74ce9b6de32b service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:01:09.984632 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1262/2490] 10.222.0.33 () {60 vars in 1396 bytes} [Sun Oct 9 01:01:09 2022] GET /identity/v3/auth/tokens?allow_expired=1 => generated 3270 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 01:01:19.980416 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=6 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:01:20.124114 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c37e40b0-3571-43a0-ba9f-457448f90795 service neutron] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:01:20.125527 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-c37e40b0-3571-43a0-ba9f-457448f90795 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:01:20.139476 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c37e40b0-3571-43a0-ba9f-457448f90795 service neutron] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:01:20.145604 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a89de32d-63e9-4e36-b1f6-ca195db3f516 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a5d95969e74157bb2d2a57207e1651', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:01:20.146186 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a89de32d-63e9-4e36-b1f6-ca195db3f516 service nova] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:01:20.146468 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a89de32d-63e9-4e36-b1f6-ca195db3f516 service nova] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:01:20.146739 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a89de32d-63e9-4e36-b1f6-ca195db3f516 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:01:20.148420 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-a89de32d-63e9-4e36-b1f6-ca195db3f516 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:01:20.158630 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a89de32d-63e9-4e36-b1f6-ca195db3f516 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:01:20.160240 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a89de32d-63e9-4e36-b1f6-ca195db3f516 service nova] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:01:20.168993 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1229/2491] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 01:01:20 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 01:01:22.560559 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4aa0afe-a255-4ddf-90b4-74ce9b6de32b service neutron] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:01:22.563220 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-c4aa0afe-a255-4ddf-90b4-74ce9b6de32b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:01:22.605067 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4aa0afe-a255-4ddf-90b4-74ce9b6de32b service neutron] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:01:22.611553 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82b5704e-a17e-46f3-9f8f-c168c7850327 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a5d95969e74157bb2d2a57207e1651', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:01:22.612192 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b5704e-a17e-46f3-9f8f-c168c7850327 service nova] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:01:22.612472 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b5704e-a17e-46f3-9f8f-c168c7850327 service nova] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:01:22.612743 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b5704e-a17e-46f3-9f8f-c168c7850327 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:01:22.615094 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-82b5704e-a17e-46f3-9f8f-c168c7850327 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:01:22.646132 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82b5704e-a17e-46f3-9f8f-c168c7850327 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:01:22.647814 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82b5704e-a17e-46f3-9f8f-c168c7850327 service nova] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:01:22.662054 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1263/2492] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 01:01:22 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 01:01:28.315910 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65044c75-d42c-4e61-96c3-1b6e5837b83a None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:01:28.316420 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65044c75-d42c-4e61-96c3-1b6e5837b83a None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:01:28.316505 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65044c75-d42c-4e61-96c3-1b6e5837b83a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:01:28.337473 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-65044c75-d42c-4e61-96c3-1b6e5837b83a None None] MFA Rules not processed for user `04bc22d558c2466ba43a456d08bdd6bd`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:01:28.338815 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-65044c75-d42c-4e61-96c3-1b6e5837b83a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:01:28.406988 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1230/2493] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:01:28 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:01:28.734549 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82b5704e-a17e-46f3-9f8f-c168c7850327 service nova] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:01:28.744428 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82b5704e-a17e-46f3-9f8f-c168c7850327 service nova] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:01:28.748440 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2774d019-03d8-4d1f-95bd-a79029156794 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:01:28.749108 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2774d019-03d8-4d1f-95bd-a79029156794 service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:01:28.749460 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2774d019-03d8-4d1f-95bd-a79029156794 service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:01:28.749806 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2774d019-03d8-4d1f-95bd-a79029156794 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:01:28.751687 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-2774d019-03d8-4d1f-95bd-a79029156794 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:01:28.761652 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2774d019-03d8-4d1f-95bd-a79029156794 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:01:28.763163 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2774d019-03d8-4d1f-95bd-a79029156794 service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:01:28.772659 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1264/2494] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 01:01:28 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 01:01:29.225068 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65044c75-d42c-4e61-96c3-1b6e5837b83a None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:01:29.226465 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-65044c75-d42c-4e61-96c3-1b6e5837b83a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:01:29.296572 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65044c75-d42c-4e61-96c3-1b6e5837b83a None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:01:29.301497 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb5d43b9-9d90-4951-a75a-3bb3345dddc7 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd25e240cb93643ed8122ccac9586936f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:01:29.302094 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb5d43b9-9d90-4951-a75a-3bb3345dddc7 service placement] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:01:29.302375 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb5d43b9-9d90-4951-a75a-3bb3345dddc7 service placement] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:01:29.302649 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb5d43b9-9d90-4951-a75a-3bb3345dddc7 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:01:29.304382 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-eb5d43b9-9d90-4951-a75a-3bb3345dddc7 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:01:29.316611 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb5d43b9-9d90-4951-a75a-3bb3345dddc7 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:01:29.318758 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb5d43b9-9d90-4951-a75a-3bb3345dddc7 service placement] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:01:29.328469 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1231/2495] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 01:01:29 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 01:01:29.622836 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2774d019-03d8-4d1f-95bd-a79029156794 service neutron] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:01:29.632640 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2774d019-03d8-4d1f-95bd-a79029156794 service neutron] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:01:29.637503 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceb5268e-7d5e-4b29-8328-a29552979a1a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:01:29.638191 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb5268e-7d5e-4b29-8328-a29552979a1a service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:01:29.638561 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb5268e-7d5e-4b29-8328-a29552979a1a service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:01:29.638878 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb5268e-7d5e-4b29-8328-a29552979a1a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:01:29.640834 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-ceb5268e-7d5e-4b29-8328-a29552979a1a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:01:29.709718 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceb5268e-7d5e-4b29-8328-a29552979a1a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:01:29.711156 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceb5268e-7d5e-4b29-8328-a29552979a1a service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:01:29.722188 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1265/2496] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 01:01:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 01:01:32.180911 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb5d43b9-9d90-4951-a75a-3bb3345dddc7 service placement] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:01:32.193298 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb5d43b9-9d90-4951-a75a-3bb3345dddc7 service placement] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:01:32.200375 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5115338a-4f5b-468d-801b-07f879005aaa service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:01:32.203801 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5115338a-4f5b-468d-801b-07f879005aaa service neutron] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:01:32.204386 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5115338a-4f5b-468d-801b-07f879005aaa service neutron] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:01:32.205121 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5115338a-4f5b-468d-801b-07f879005aaa service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:01:32.208201 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-5115338a-4f5b-468d-801b-07f879005aaa service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:01:32.219617 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5115338a-4f5b-468d-801b-07f879005aaa service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:01:32.221701 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5115338a-4f5b-468d-801b-07f879005aaa service neutron] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:01:32.238913 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1232/2497] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 01:01:32 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 59 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 01:01:34.206113 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceb5268e-7d5e-4b29-8328-a29552979a1a service neutron] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:01:34.207635 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-ceb5268e-7d5e-4b29-8328-a29552979a1a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:01:34.219400 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceb5268e-7d5e-4b29-8328-a29552979a1a service neutron] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:01:34.224775 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bc18275-8d97-47d9-a42b-35cbd9b0ecce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:01:34.225403 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc18275-8d97-47d9-a42b-35cbd9b0ecce admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:01:34.225863 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc18275-8d97-47d9-a42b-35cbd9b0ecce admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:01:34.226288 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc18275-8d97-47d9-a42b-35cbd9b0ecce admin admin] PATH_INFO: `/v3/users/438446d1e6b3402bb18b547c089391f8` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:01:34.227572 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bc18275-8d97-47d9-a42b-35cbd9b0ecce admin admin] RBAC: Authorizing `identity:delete_user(user_id=438446d1e6b3402bb18b547c089391f8)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:01:34.229664 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bc18275-8d97-47d9-a42b-35cbd9b0ecce admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:01:34.283814 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-0bc18275-8d97-47d9-a42b-35cbd9b0ecce admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '438446d1e6b3402bb18b547c089391f8'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:01:34.291348 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-0bc18275-8d97-47d9-a42b-35cbd9b0ecce admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '438446d1e6b3402bb18b547c089391f8'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:01:34.292023 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-0bc18275-8d97-47d9-a42b-35cbd9b0ecce admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '438446d1e6b3402bb18b547c089391f8'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:01:34.292626 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-0bc18275-8d97-47d9-a42b-35cbd9b0ecce admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '438446d1e6b3402bb18b547c089391f8'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:01:34.299668 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-0bc18275-8d97-47d9-a42b-35cbd9b0ecce admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '438446d1e6b3402bb18b547c089391f8'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:01:34.317190 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1266/2498] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 01:01:34 2022] DELETE /identity/v3/users/438446d1e6b3402bb18b547c089391f8 => generated 0 bytes in 112 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:01:34.483932 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5115338a-4f5b-468d-801b-07f879005aaa service neutron] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:01:34.496317 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5115338a-4f5b-468d-801b-07f879005aaa service neutron] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:01:34.553593 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c46fc1c-20ef-4e55-abeb-dd24c07cb41b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:01:34.554221 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c46fc1c-20ef-4e55-abeb-dd24c07cb41b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:01:34.554503 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c46fc1c-20ef-4e55-abeb-dd24c07cb41b admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:01:34.554776 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c46fc1c-20ef-4e55-abeb-dd24c07cb41b admin admin] PATH_INFO: `/v3/projects/f68941040b3049bd9c62e2fc061d94c1` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:01:34.555925 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c46fc1c-20ef-4e55-abeb-dd24c07cb41b admin admin] RBAC: Authorizing `identity:delete_project(project_id=f68941040b3049bd9c62e2fc061d94c1)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:01:34.557831 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c46fc1c-20ef-4e55-abeb-dd24c07cb41b admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:01:34.568111 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:01:34.604231 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-0c46fc1c-20ef-4e55-abeb-dd24c07cb41b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f68941040b3049bd9c62e2fc061d94c1', 'request_id': 'req-0c46fc1c-20ef-4e55-abeb-dd24c07cb41b'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:01:34.610438 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-0c46fc1c-20ef-4e55-abeb-dd24c07cb41b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f68941040b3049bd9c62e2fc061d94c1', 'request_id': 'req-0c46fc1c-20ef-4e55-abeb-dd24c07cb41b'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:01:34.617285 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-0c46fc1c-20ef-4e55-abeb-dd24c07cb41b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f68941040b3049bd9c62e2fc061d94c1', 'request_id': 'req-0c46fc1c-20ef-4e55-abeb-dd24c07cb41b'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:01:34.632594 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-0c46fc1c-20ef-4e55-abeb-dd24c07cb41b admin admin] The token cache is being invalidate because project f68941040b3049bd9c62e2fc061d94c1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81453) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 01:01:34.632881 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-0c46fc1c-20ef-4e55-abeb-dd24c07cb41b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:01:34.633546 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-0c46fc1c-20ef-4e55-abeb-dd24c07cb41b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:01:34.634999 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1233/2499] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 01:01:34 2022] DELETE /identity/v3/projects/f68941040b3049bd9c62e2fc061d94c1 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:01:34.640048 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff48862-ca2e-4351-8703-cca5d0b7f1bd None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:01:34.640337 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff48862-ca2e-4351-8703-cca5d0b7f1bd None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:01:34.640612 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff48862-ca2e-4351-8703-cca5d0b7f1bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:01:34.660359 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-8ff48862-ca2e-4351-8703-cca5d0b7f1bd None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:01:34.662131 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-8ff48862-ca2e-4351-8703-cca5d0b7f1bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:01:34.693998 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:01:34.721312 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1267/2500] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:01:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:01:34.729569 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e38b935-5b06-4464-aaab-5cda165efac5 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:01:34.729855 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e38b935-5b06-4464-aaab-5cda165efac5 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:01:34.730131 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e38b935-5b06-4464-aaab-5cda165efac5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:01:34.750575 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-1e38b935-5b06-4464-aaab-5cda165efac5 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:01:34.751802 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-1e38b935-5b06-4464-aaab-5cda165efac5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:01:34.759616 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1234/2501] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:01:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 09 01:01:34.763666 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ff48862-ca2e-4351-8703-cca5d0b7f1bd None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:01:34.773703 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ff48862-ca2e-4351-8703-cca5d0b7f1bd None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:01:34.778481 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbb45254-046f-4f01-935a-84ad568f617d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:01:34.779074 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb45254-046f-4f01-935a-84ad568f617d admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:01:34.779353 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb45254-046f-4f01-935a-84ad568f617d admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:01:34.779652 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb45254-046f-4f01-935a-84ad568f617d admin admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:01:34.780292 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbb45254-046f-4f01-935a-84ad568f617d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 01:01:34.780595 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbb45254-046f-4f01-935a-84ad568f617d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:01:34.782176 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbb45254-046f-4f01-935a-84ad568f617d admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:01:34.794196 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1268/2502] 10.222.0.33 () {58 vars in 1153 bytes} [Sun Oct 9 01:01:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 01:01:34.798339 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e38b935-5b06-4464-aaab-5cda165efac5 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:01:34.807867 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e38b935-5b06-4464-aaab-5cda165efac5 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:01:34.812598 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d255dd11-6f55-41bd-b00b-b1d8885a6dff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:01:34.813183 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d255dd11-6f55-41bd-b00b-b1d8885a6dff admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:01:34.813664 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d255dd11-6f55-41bd-b00b-b1d8885a6dff admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:01:34.813938 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d255dd11-6f55-41bd-b00b-b1d8885a6dff admin admin] PATH_INFO: `/v3/projects` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:01:34.814569 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d255dd11-6f55-41bd-b00b-b1d8885a6dff admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:01:34.816499 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d255dd11-6f55-41bd-b00b-b1d8885a6dff admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:01:34.845256 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1235/2503] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 01:01:34 2022] POST /identity/v3/projects => generated 387 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 01:01:34.849376 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbb45254-046f-4f01-935a-84ad568f617d admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:01:34.859379 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbb45254-046f-4f01-935a-84ad568f617d admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:01:34.919737 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30db9557-9963-4f9b-a73f-a9c9ce25e7f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:01:34.920345 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30db9557-9963-4f9b-a73f-a9c9ce25e7f3 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:01:34.920624 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30db9557-9963-4f9b-a73f-a9c9ce25e7f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:01:34.920897 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30db9557-9963-4f9b-a73f-a9c9ce25e7f3 admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:01:34.921536 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30db9557-9963-4f9b-a73f-a9c9ce25e7f3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:01:34.923470 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30db9557-9963-4f9b-a73f-a9c9ce25e7f3 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:01:34.945713 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1269/2504] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 01:01:34 2022] POST /identity/v3/users => generated 294 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 09 01:01:34.950083 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d255dd11-6f55-41bd-b00b-b1d8885a6dff admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:01:34.963807 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d255dd11-6f55-41bd-b00b-b1d8885a6dff admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:01:34.968552 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94bcf9da-ed86-4327-90cc-39e21c69eade admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:01:34.969144 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94bcf9da-ed86-4327-90cc-39e21c69eade admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:01:34.969421 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94bcf9da-ed86-4327-90cc-39e21c69eade admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:01:34.969688 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94bcf9da-ed86-4327-90cc-39e21c69eade admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:01:34.970311 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94bcf9da-ed86-4327-90cc-39e21c69eade admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:01:34.971995 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94bcf9da-ed86-4327-90cc-39e21c69eade admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:01:34.982987 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1236/2505] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 01:01:34 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 01:01:34.986965 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30db9557-9963-4f9b-a73f-a9c9ce25e7f3 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:01:34.997611 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30db9557-9963-4f9b-a73f-a9c9ce25e7f3 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:01:35.002233 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c662dac7-0f8d-4568-a120-b1d30f5d1d6b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:01:35.002821 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c662dac7-0f8d-4568-a120-b1d30f5d1d6b admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:01:35.003103 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c662dac7-0f8d-4568-a120-b1d30f5d1d6b admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:01:35.003379 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c662dac7-0f8d-4568-a120-b1d30f5d1d6b admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:01:35.004055 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c662dac7-0f8d-4568-a120-b1d30f5d1d6b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:01:35.005638 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c662dac7-0f8d-4568-a120-b1d30f5d1d6b admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:01:35.015943 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1270/2506] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 01:01:34 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 01:01:35.020074 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94bcf9da-ed86-4327-90cc-39e21c69eade admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:01:35.030074 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94bcf9da-ed86-4327-90cc-39e21c69eade admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:01:35.034606 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa184eeb-242b-40a8-969a-d6b5448e8686 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:01:35.035237 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa184eeb-242b-40a8-969a-d6b5448e8686 admin admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:01:35.036069 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa184eeb-242b-40a8-969a-d6b5448e8686 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:01:35.036437 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa184eeb-242b-40a8-969a-d6b5448e8686 admin admin] PATH_INFO: `/v3/projects/410212da4a794c07b30ed22cc5b7e760/users/56b5d764f0044358ac594a2edd6f4c0f/roles/588016933bab436d8b7c3ac76827356d` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:01:35.050737 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa184eeb-242b-40a8-969a-d6b5448e8686 admin admin] RBAC: Authorizing `identity:create_grant(project_id=410212da4a794c07b30ed22cc5b7e760, user_id=56b5d764f0044358ac594a2edd6f4c0f, role_id=588016933bab436d8b7c3ac76827356d)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:01:35.054079 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa184eeb-242b-40a8-969a-d6b5448e8686 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:01:35.066710 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1237/2507] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 01:01:35 2022] PUT /identity/v3/projects/410212da4a794c07b30ed22cc5b7e760/users/56b5d764f0044358ac594a2edd6f4c0f/roles/588016933bab436d8b7c3ac76827356d => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 09 01:01:35.071830 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c62c3424-2247-4cf2-9059-3284ec2f9040 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:01:35.072124 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c62c3424-2247-4cf2-9059-3284ec2f9040 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:01:35.072407 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c62c3424-2247-4cf2-9059-3284ec2f9040 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:01:35.091124 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-c62c3424-2247-4cf2-9059-3284ec2f9040 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:01:35.092358 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-c62c3424-2247-4cf2-9059-3284ec2f9040 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:01:35.149655 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1271/2508] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:01:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:01:35.159870 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa184eeb-242b-40a8-969a-d6b5448e8686 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:01:35.162170 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-fa184eeb-242b-40a8-969a-d6b5448e8686 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:01:35.217530 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa184eeb-242b-40a8-969a-d6b5448e8686 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:01:35.221094 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bc5558c-2e7d-4ae1-ae8b-dae6c9c09cc6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:01:35.221695 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc5558c-2e7d-4ae1-ae8b-dae6c9c09cc6 service neutron] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:01:35.221978 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc5558c-2e7d-4ae1-ae8b-dae6c9c09cc6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:01:35.222253 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc5558c-2e7d-4ae1-ae8b-dae6c9c09cc6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:01:35.231584 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bc5558c-2e7d-4ae1-ae8b-dae6c9c09cc6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:01:35.233020 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bc5558c-2e7d-4ae1-ae8b-dae6c9c09cc6 service neutron] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:01:35.243368 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1238/2509] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 01:01:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 01:01:39.079956 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f2165be-7744-43cf-8fa0-8af1b91c2a98 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:01:39.080496 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f2165be-7744-43cf-8fa0-8af1b91c2a98 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:01:39.080729 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f2165be-7744-43cf-8fa0-8af1b91c2a98 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:01:39.114863 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-9f2165be-7744-43cf-8fa0-8af1b91c2a98 None None] MFA Rules not processed for user `56b5d764f0044358ac594a2edd6f4c0f`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:01:39.116261 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-9f2165be-7744-43cf-8fa0-8af1b91c2a98 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:01:39.154691 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:01:39.205743 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1272/2510] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:01:39 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:01:39.221573 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bc5558c-2e7d-4ae1-ae8b-dae6c9c09cc6 service neutron] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:01:39.223112 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-3bc5558c-2e7d-4ae1-ae8b-dae6c9c09cc6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:01:39.280153 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:01:39.319020 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bc5558c-2e7d-4ae1-ae8b-dae6c9c09cc6 service neutron] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:01:39.325430 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0619724c-0a3b-4d79-8515-8cc4bfa36d0b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a5d95969e74157bb2d2a57207e1651', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:01:39.326510 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0619724c-0a3b-4d79-8515-8cc4bfa36d0b service nova] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:01:39.326510 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0619724c-0a3b-4d79-8515-8cc4bfa36d0b service nova] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:01:39.326721 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0619724c-0a3b-4d79-8515-8cc4bfa36d0b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:01:39.335473 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0619724c-0a3b-4d79-8515-8cc4bfa36d0b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:01:39.337127 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0619724c-0a3b-4d79-8515-8cc4bfa36d0b service nova] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:01:39.346904 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1239/2511] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 01:01:39 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 126 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 01:01:40.782634 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45692734-a9e2-4c31-ac20-ad8af394203d None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:01:40.783158 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45692734-a9e2-4c31-ac20-ad8af394203d None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:01:40.783257 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45692734-a9e2-4c31-ac20-ad8af394203d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:01:40.803146 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-45692734-a9e2-4c31-ac20-ad8af394203d None None] MFA Rules not processed for user `56b5d764f0044358ac594a2edd6f4c0f`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:01:40.804469 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-45692734-a9e2-4c31-ac20-ad8af394203d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:01:40.812058 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1273/2512] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:01:40 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:01:44.142824 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0619724c-0a3b-4d79-8515-8cc4bfa36d0b service nova] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:01:44.154223 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0619724c-0a3b-4d79-8515-8cc4bfa36d0b service nova] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:01:44.161307 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58c4c12f-d6ce-44d4-9bbc-3c902214ae57 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a5d95969e74157bb2d2a57207e1651', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:01:44.161912 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c4c12f-d6ce-44d4-9bbc-3c902214ae57 service nova] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:01:44.162192 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c4c12f-d6ce-44d4-9bbc-3c902214ae57 service nova] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:01:44.162467 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c4c12f-d6ce-44d4-9bbc-3c902214ae57 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:01:44.164240 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-58c4c12f-d6ce-44d4-9bbc-3c902214ae57 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:01:44.223357 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58c4c12f-d6ce-44d4-9bbc-3c902214ae57 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:01:44.224978 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58c4c12f-d6ce-44d4-9bbc-3c902214ae57 service nova] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:01:44.234495 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1240/2513] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 01:01:44 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 01:01:50.796862 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=14 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:01:54.001240 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45692734-a9e2-4c31-ac20-ad8af394203d None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:01:54.002779 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-45692734-a9e2-4c31-ac20-ad8af394203d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:01:54.019156 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45692734-a9e2-4c31-ac20-ad8af394203d None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:01:54.024622 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b8bbf62-d7cd-4ffb-8b38-4772500b33df service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a5d95969e74157bb2d2a57207e1651', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:01:54.025252 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b8bbf62-d7cd-4ffb-8b38-4772500b33df service nova] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:01:54.025531 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b8bbf62-d7cd-4ffb-8b38-4772500b33df service nova] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:01:54.026052 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b8bbf62-d7cd-4ffb-8b38-4772500b33df service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:01:54.027807 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-2b8bbf62-d7cd-4ffb-8b38-4772500b33df service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:01:54.040939 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b8bbf62-d7cd-4ffb-8b38-4772500b33df service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:01:54.042550 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b8bbf62-d7cd-4ffb-8b38-4772500b33df service nova] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:01:54.054057 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1274/2514] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 01:01:54 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 53 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 01:01:54.229986 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=45 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:01:55.577507 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58c4c12f-d6ce-44d4-9bbc-3c902214ae57 service nova] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:01:55.591663 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58c4c12f-d6ce-44d4-9bbc-3c902214ae57 service nova] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:01:55.595898 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f137775-cd13-4343-a164-47667426462b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:01:55.596577 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f137775-cd13-4343-a164-47667426462b service neutron] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:01:55.596928 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f137775-cd13-4343-a164-47667426462b service neutron] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:01:55.597271 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f137775-cd13-4343-a164-47667426462b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:01:55.600552 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-8f137775-cd13-4343-a164-47667426462b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:01:55.613489 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f137775-cd13-4343-a164-47667426462b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:01:55.614934 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f137775-cd13-4343-a164-47667426462b service neutron] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:01:55.624579 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1241/2515] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 01:01:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 01:01:58.238492 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b8bbf62-d7cd-4ffb-8b38-4772500b33df service nova] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:01:58.255561 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b8bbf62-d7cd-4ffb-8b38-4772500b33df service nova] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:01:58.262270 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e276ac7d-cdaa-46cc-b472-c499501534b9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a5d95969e74157bb2d2a57207e1651', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:01:58.263826 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e276ac7d-cdaa-46cc-b472-c499501534b9 service nova] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:01:58.264338 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e276ac7d-cdaa-46cc-b472-c499501534b9 service nova] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:01:58.264932 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e276ac7d-cdaa-46cc-b472-c499501534b9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:01:58.267388 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-e276ac7d-cdaa-46cc-b472-c499501534b9 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:01:58.289247 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e276ac7d-cdaa-46cc-b472-c499501534b9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:01:58.293537 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e276ac7d-cdaa-46cc-b472-c499501534b9 service nova] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:01:58.315433 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1275/2516] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 01:01:58 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 78 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 01:02:00.026088 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f137775-cd13-4343-a164-47667426462b service neutron] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:00.028151 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-8f137775-cd13-4343-a164-47667426462b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:00.040689 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f137775-cd13-4343-a164-47667426462b service neutron] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:00.045691 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be831ebc-36bc-4029-8248-7ee4b8b36530 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:00.046448 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be831ebc-36bc-4029-8248-7ee4b8b36530 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:00.046730 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be831ebc-36bc-4029-8248-7ee4b8b36530 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:00.047001 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be831ebc-36bc-4029-8248-7ee4b8b36530 admin admin] PATH_INFO: `/v3/users/1512c91201f14b75a9be35b89c2097ef` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:00.048236 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be831ebc-36bc-4029-8248-7ee4b8b36530 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1512c91201f14b75a9be35b89c2097ef)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:00.050148 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be831ebc-36bc-4029-8248-7ee4b8b36530 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:00.120558 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-be831ebc-36bc-4029-8248-7ee4b8b36530 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1512c91201f14b75a9be35b89c2097ef'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:00.129450 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-be831ebc-36bc-4029-8248-7ee4b8b36530 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1512c91201f14b75a9be35b89c2097ef'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:00.130290 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-be831ebc-36bc-4029-8248-7ee4b8b36530 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1512c91201f14b75a9be35b89c2097ef'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:00.138754 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-be831ebc-36bc-4029-8248-7ee4b8b36530 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1512c91201f14b75a9be35b89c2097ef'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:00.151715 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-be831ebc-36bc-4029-8248-7ee4b8b36530 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1512c91201f14b75a9be35b89c2097ef'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:00.157672 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1242/2517] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 01:02:00 2022] DELETE /identity/v3/users/1512c91201f14b75a9be35b89c2097ef => generated 0 bytes in 132 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:02:00.339822 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e276ac7d-cdaa-46cc-b472-c499501534b9 service nova] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:00.364263 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e276ac7d-cdaa-46cc-b472-c499501534b9 service nova] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:00.450816 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18fb80cf-8780-45c7-b232-9b173b1beff6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:00.451459 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18fb80cf-8780-45c7-b232-9b173b1beff6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:00.452138 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18fb80cf-8780-45c7-b232-9b173b1beff6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:00.452455 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18fb80cf-8780-45c7-b232-9b173b1beff6 admin admin] PATH_INFO: `/v3/projects/40e631c5e320489cac9cea85118c6263` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:00.453602 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18fb80cf-8780-45c7-b232-9b173b1beff6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=40e631c5e320489cac9cea85118c6263)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:00.456082 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18fb80cf-8780-45c7-b232-9b173b1beff6 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:00.517334 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-18fb80cf-8780-45c7-b232-9b173b1beff6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '40e631c5e320489cac9cea85118c6263', 'request_id': 'req-18fb80cf-8780-45c7-b232-9b173b1beff6'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:00.527493 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-18fb80cf-8780-45c7-b232-9b173b1beff6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '40e631c5e320489cac9cea85118c6263', 'request_id': 'req-18fb80cf-8780-45c7-b232-9b173b1beff6'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:00.537663 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-18fb80cf-8780-45c7-b232-9b173b1beff6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '40e631c5e320489cac9cea85118c6263', 'request_id': 'req-18fb80cf-8780-45c7-b232-9b173b1beff6'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:00.558673 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-18fb80cf-8780-45c7-b232-9b173b1beff6 admin admin] The token cache is being invalidate because project 40e631c5e320489cac9cea85118c6263 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 01:02:00.559214 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-18fb80cf-8780-45c7-b232-9b173b1beff6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:00.560067 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-18fb80cf-8780-45c7-b232-9b173b1beff6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:00.561835 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1276/2518] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 01:02:00 2022] DELETE /identity/v3/projects/40e631c5e320489cac9cea85118c6263 => generated 0 bytes in 223 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:02:00.566527 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64ff338f-9708-4264-b3bb-f31d8cff8b16 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:00.566815 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64ff338f-9708-4264-b3bb-f31d8cff8b16 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:00.567092 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64ff338f-9708-4264-b3bb-f31d8cff8b16 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:00.588506 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-64ff338f-9708-4264-b3bb-f31d8cff8b16 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:02:00.589936 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-64ff338f-9708-4264-b3bb-f31d8cff8b16 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:00.671183 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1243/2519] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:02:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:02:00.680514 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c255cf49-a2c3-44d0-923d-0973ec7b5e7f None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:00.680832 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c255cf49-a2c3-44d0-923d-0973ec7b5e7f None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:00.681111 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c255cf49-a2c3-44d0-923d-0973ec7b5e7f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:00.702960 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-c255cf49-a2c3-44d0-923d-0973ec7b5e7f None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:02:00.704350 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-c255cf49-a2c3-44d0-923d-0973ec7b5e7f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:00.713616 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1277/2520] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:02:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 09 01:02:00.717821 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64ff338f-9708-4264-b3bb-f31d8cff8b16 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:00.728550 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64ff338f-9708-4264-b3bb-f31d8cff8b16 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:00.733524 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-904dfdd5-0b5c-4533-93d0-f864bef5e8cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:00.734388 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-904dfdd5-0b5c-4533-93d0-f864bef5e8cc admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:00.734734 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-904dfdd5-0b5c-4533-93d0-f864bef5e8cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:00.735125 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-904dfdd5-0b5c-4533-93d0-f864bef5e8cc admin admin] PATH_INFO: `/v3/domains` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:00.735914 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-904dfdd5-0b5c-4533-93d0-f864bef5e8cc admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 01:02:00.736341 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-904dfdd5-0b5c-4533-93d0-f864bef5e8cc admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:00.738819 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-904dfdd5-0b5c-4533-93d0-f864bef5e8cc admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:00.751298 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1244/2521] 10.222.0.33 () {58 vars in 1153 bytes} [Sun Oct 9 01:02:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 01:02:00.755690 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c255cf49-a2c3-44d0-923d-0973ec7b5e7f None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:00.764858 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c255cf49-a2c3-44d0-923d-0973ec7b5e7f None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:00.769479 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aed19b74-bc4d-43fa-b66f-acbbab75026c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:00.770067 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed19b74-bc4d-43fa-b66f-acbbab75026c admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:00.770345 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed19b74-bc4d-43fa-b66f-acbbab75026c admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:00.770614 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed19b74-bc4d-43fa-b66f-acbbab75026c admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:00.771233 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aed19b74-bc4d-43fa-b66f-acbbab75026c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:00.773225 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aed19b74-bc4d-43fa-b66f-acbbab75026c admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:00.801401 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1278/2522] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 01:02:00 2022] POST /identity/v3/projects => generated 403 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 01:02:00.805464 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-904dfdd5-0b5c-4533-93d0-f864bef5e8cc admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:00.815575 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-904dfdd5-0b5c-4533-93d0-f864bef5e8cc admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:00.866201 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:02:00.881257 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2496047-f0dc-4b85-af2c-0d259cade474 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:00.881947 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2496047-f0dc-4b85-af2c-0d259cade474 admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:00.882287 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2496047-f0dc-4b85-af2c-0d259cade474 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:00.882624 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2496047-f0dc-4b85-af2c-0d259cade474 admin admin] PATH_INFO: `/v3/users` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:00.883326 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2496047-f0dc-4b85-af2c-0d259cade474 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:00.885689 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2496047-f0dc-4b85-af2c-0d259cade474 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:00.915021 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1245/2523] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 01:02:00 2022] POST /identity/v3/users => generated 302 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 01:02:00.918899 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aed19b74-bc4d-43fa-b66f-acbbab75026c admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:00.932585 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aed19b74-bc4d-43fa-b66f-acbbab75026c admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:00.945339 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18460c95-df7a-4276-891e-5af3aa2058c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:00.946911 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18460c95-df7a-4276-891e-5af3aa2058c8 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:00.947377 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18460c95-df7a-4276-891e-5af3aa2058c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:00.947853 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18460c95-df7a-4276-891e-5af3aa2058c8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:00.948858 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18460c95-df7a-4276-891e-5af3aa2058c8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:00.951383 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18460c95-df7a-4276-891e-5af3aa2058c8 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:00.970152 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1279/2524] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 01:02:00 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 01:02:00.974526 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2496047-f0dc-4b85-af2c-0d259cade474 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:00.986343 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2496047-f0dc-4b85-af2c-0d259cade474 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:00.994891 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-266cbfdf-851d-4b9f-ad26-fcacbda427b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:00.995516 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-266cbfdf-851d-4b9f-ad26-fcacbda427b0 admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:00.995833 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-266cbfdf-851d-4b9f-ad26-fcacbda427b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:00.996110 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-266cbfdf-851d-4b9f-ad26-fcacbda427b0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:00.996758 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-266cbfdf-851d-4b9f-ad26-fcacbda427b0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:00.998365 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-266cbfdf-851d-4b9f-ad26-fcacbda427b0 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:01.015667 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1246/2525] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 01:02:00 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 01:02:01.017318 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18460c95-df7a-4276-891e-5af3aa2058c8 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:01.030917 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18460c95-df7a-4276-891e-5af3aa2058c8 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:01.038563 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4902e5d1-dd33-4e44-9933-b02b620308cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:01.039327 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4902e5d1-dd33-4e44-9933-b02b620308cd admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:01.039670 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4902e5d1-dd33-4e44-9933-b02b620308cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:01.040083 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4902e5d1-dd33-4e44-9933-b02b620308cd admin admin] PATH_INFO: `/v3/projects/645b73a7f71d47c891fae7637fe36773/users/ca723f1977724f6d97fb569c918720bf/roles/588016933bab436d8b7c3ac76827356d` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:01.059272 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4902e5d1-dd33-4e44-9933-b02b620308cd admin admin] RBAC: Authorizing `identity:create_grant(project_id=645b73a7f71d47c891fae7637fe36773, user_id=ca723f1977724f6d97fb569c918720bf, role_id=588016933bab436d8b7c3ac76827356d)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:01.062047 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4902e5d1-dd33-4e44-9933-b02b620308cd admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:01.072685 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:02:01.087024 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1280/2526] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 01:02:01 2022] PUT /identity/v3/projects/645b73a7f71d47c891fae7637fe36773/users/ca723f1977724f6d97fb569c918720bf/roles/588016933bab436d8b7c3ac76827356d => generated 0 bytes in 67 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:02:01.089208 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85573e91-f1b3-4bfc-851f-88bd3bfe505f None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:01.089677 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85573e91-f1b3-4bfc-851f-88bd3bfe505f None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:01.090118 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85573e91-f1b3-4bfc-851f-88bd3bfe505f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:01.113124 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-85573e91-f1b3-4bfc-851f-88bd3bfe505f None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:02:01.115222 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-85573e91-f1b3-4bfc-851f-88bd3bfe505f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:01.193439 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1247/2527] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:02:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:02:01.204200 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4902e5d1-dd33-4e44-9933-b02b620308cd admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:01.206205 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-4902e5d1-dd33-4e44-9933-b02b620308cd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:01.267683 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4902e5d1-dd33-4e44-9933-b02b620308cd admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:01.271508 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fbde8de-25fb-4940-b94d-d30efbdd9940 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:01.272196 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fbde8de-25fb-4940-b94d-d30efbdd9940 service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:01.272480 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fbde8de-25fb-4940-b94d-d30efbdd9940 service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:01.272756 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fbde8de-25fb-4940-b94d-d30efbdd9940 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:01.282996 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fbde8de-25fb-4940-b94d-d30efbdd9940 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:01.284489 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fbde8de-25fb-4940-b94d-d30efbdd9940 service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:01.295343 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1281/2528] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 01:02:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 01:02:01.915955 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac2017a0-6533-485d-b107-78d0b64d2835 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:01.916242 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac2017a0-6533-485d-b107-78d0b64d2835 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:01.916516 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac2017a0-6533-485d-b107-78d0b64d2835 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:01.944963 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-ac2017a0-6533-485d-b107-78d0b64d2835 None None] MFA Rules not processed for user `ca723f1977724f6d97fb569c918720bf`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:02:01.946197 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-ac2017a0-6533-485d-b107-78d0b64d2835 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:02.016655 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1248/2529] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:02:01 2022] POST /identity/v3/auth/tokens => generated 3280 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:02:02.022018 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fbde8de-25fb-4940-b94d-d30efbdd9940 service neutron] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:02.031994 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fbde8de-25fb-4940-b94d-d30efbdd9940 service neutron] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:02.036682 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc246c14-e8c5-4af4-a5c5-982981df8ad2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:02.037347 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc246c14-e8c5-4af4-a5c5-982981df8ad2 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:02.037757 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc246c14-e8c5-4af4-a5c5-982981df8ad2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:02.038050 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc246c14-e8c5-4af4-a5c5-982981df8ad2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:02.038767 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc246c14-e8c5-4af4-a5c5-982981df8ad2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:02.040805 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc246c14-e8c5-4af4-a5c5-982981df8ad2 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:02.076520 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1282/2530] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 01:02:02 2022] POST /identity/v3/projects => generated 403 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 01:02:02.078574 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac2017a0-6533-485d-b107-78d0b64d2835 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:02.089921 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac2017a0-6533-485d-b107-78d0b64d2835 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:02.153075 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:02:02.156292 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24b7d8d2-7bbc-41d1-8789-2e125807ea4b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:02.156957 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b7d8d2-7bbc-41d1-8789-2e125807ea4b admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:02.157515 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b7d8d2-7bbc-41d1-8789-2e125807ea4b admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:02.158018 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b7d8d2-7bbc-41d1-8789-2e125807ea4b admin admin] PATH_INFO: `/v3/users` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:02.159268 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24b7d8d2-7bbc-41d1-8789-2e125807ea4b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:02.161513 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24b7d8d2-7bbc-41d1-8789-2e125807ea4b admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:02.186459 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1249/2531] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 01:02:02 2022] POST /identity/v3/users => generated 308 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 01:02:02.188075 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc246c14-e8c5-4af4-a5c5-982981df8ad2 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:02.199310 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc246c14-e8c5-4af4-a5c5-982981df8ad2 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:02.204202 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b10ff686-8484-4019-8988-ddce6558e263 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:02.204946 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10ff686-8484-4019-8988-ddce6558e263 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:02.205379 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10ff686-8484-4019-8988-ddce6558e263 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:02.205716 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10ff686-8484-4019-8988-ddce6558e263 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:02.206425 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b10ff686-8484-4019-8988-ddce6558e263 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:02.208718 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b10ff686-8484-4019-8988-ddce6558e263 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:02.223247 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1283/2532] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 01:02:02 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 01:02:02.227540 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24b7d8d2-7bbc-41d1-8789-2e125807ea4b admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:02.238058 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24b7d8d2-7bbc-41d1-8789-2e125807ea4b admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:02.242814 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f27c4501-4d62-41d7-84c3-09322a0d425d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:02.243471 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f27c4501-4d62-41d7-84c3-09322a0d425d admin admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:02.243864 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f27c4501-4d62-41d7-84c3-09322a0d425d admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:02.244145 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f27c4501-4d62-41d7-84c3-09322a0d425d admin admin] PATH_INFO: `/v3/projects/77cd8ace7047445fb451ab960e27f19a/users/1e2fc56fa35b49c6994cba0d4b949eee/roles/092036ee46ac427baf4d56468575f697` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:02.260270 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f27c4501-4d62-41d7-84c3-09322a0d425d admin admin] RBAC: Authorizing `identity:create_grant(project_id=77cd8ace7047445fb451ab960e27f19a, user_id=1e2fc56fa35b49c6994cba0d4b949eee, role_id=092036ee46ac427baf4d56468575f697)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:02.262957 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f27c4501-4d62-41d7-84c3-09322a0d425d admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:02.278816 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1250/2533] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 01:02:02 2022] PUT /identity/v3/projects/77cd8ace7047445fb451ab960e27f19a/users/1e2fc56fa35b49c6994cba0d4b949eee/roles/092036ee46ac427baf4d56468575f697 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 09 01:02:02.282857 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b10ff686-8484-4019-8988-ddce6558e263 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:02.293918 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b10ff686-8484-4019-8988-ddce6558e263 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:02.367911 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e3f6f6d-bacb-4d17-bf78-9c414478277e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:02.370799 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e3f6f6d-bacb-4d17-bf78-9c414478277e admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:02.371083 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e3f6f6d-bacb-4d17-bf78-9c414478277e admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:02.371362 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e3f6f6d-bacb-4d17-bf78-9c414478277e admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:02.372034 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e3f6f6d-bacb-4d17-bf78-9c414478277e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:02.373640 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e3f6f6d-bacb-4d17-bf78-9c414478277e admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:02.384923 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1284/2534] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 01:02:02 2022] GET /identity/v3/roles => generated 1420 bytes in 103 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 01:02:02.389146 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f27c4501-4d62-41d7-84c3-09322a0d425d admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:02.400074 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f27c4501-4d62-41d7-84c3-09322a0d425d admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:02.405039 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3090e6c4-0594-4444-8fa7-f0be6f220bc3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:02.405769 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3090e6c4-0594-4444-8fa7-f0be6f220bc3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:02.406139 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3090e6c4-0594-4444-8fa7-f0be6f220bc3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:02.406602 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3090e6c4-0594-4444-8fa7-f0be6f220bc3 admin admin] PATH_INFO: `/v3/projects/77cd8ace7047445fb451ab960e27f19a/users/1e2fc56fa35b49c6994cba0d4b949eee/roles/092036ee46ac427baf4d56468575f697` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:02.409002 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3090e6c4-0594-4444-8fa7-f0be6f220bc3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=77cd8ace7047445fb451ab960e27f19a, user_id=1e2fc56fa35b49c6994cba0d4b949eee, role_id=092036ee46ac427baf4d56468575f697)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:02.411723 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3090e6c4-0594-4444-8fa7-f0be6f220bc3 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:02.422458 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1251/2535] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 01:02:02 2022] PUT /identity/v3/projects/77cd8ace7047445fb451ab960e27f19a/users/1e2fc56fa35b49c6994cba0d4b949eee/roles/092036ee46ac427baf4d56468575f697 => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:02:03.138059 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f32ddf6f-4f4b-4896-b48d-338421c5b92a None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:03.138520 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f32ddf6f-4f4b-4896-b48d-338421c5b92a None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:03.138616 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f32ddf6f-4f4b-4896-b48d-338421c5b92a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:03.167723 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-f32ddf6f-4f4b-4896-b48d-338421c5b92a None None] MFA Rules not processed for user `1e2fc56fa35b49c6994cba0d4b949eee`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:02:03.169514 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-f32ddf6f-4f4b-4896-b48d-338421c5b92a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:03.223038 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:02:03.244012 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1285/2536] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:02:03 2022] POST /identity/v3/auth/tokens => generated 3347 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:02:04.276632 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3090e6c4-0594-4444-8fa7-f0be6f220bc3 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:04.348881 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3090e6c4-0594-4444-8fa7-f0be6f220bc3 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:04.353530 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08e76568-7653-4659-8038-0c9151942c22 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:04.354241 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e76568-7653-4659-8038-0c9151942c22 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:04.354587 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e76568-7653-4659-8038-0c9151942c22 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:04.354921 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e76568-7653-4659-8038-0c9151942c22 admin admin] PATH_INFO: `/v3/users/ca723f1977724f6d97fb569c918720bf` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:04.356183 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08e76568-7653-4659-8038-0c9151942c22 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ca723f1977724f6d97fb569c918720bf)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:04.358150 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08e76568-7653-4659-8038-0c9151942c22 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:04.403915 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-08e76568-7653-4659-8038-0c9151942c22 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ca723f1977724f6d97fb569c918720bf'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:04.410697 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-08e76568-7653-4659-8038-0c9151942c22 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ca723f1977724f6d97fb569c918720bf'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:04.411355 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-08e76568-7653-4659-8038-0c9151942c22 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ca723f1977724f6d97fb569c918720bf'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:04.418569 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-08e76568-7653-4659-8038-0c9151942c22 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ca723f1977724f6d97fb569c918720bf'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:04.430524 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-08e76568-7653-4659-8038-0c9151942c22 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ca723f1977724f6d97fb569c918720bf'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:04.436425 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1252/2537] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 01:02:04 2022] DELETE /identity/v3/users/ca723f1977724f6d97fb569c918720bf => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 09 01:02:04.613557 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f32ddf6f-4f4b-4896-b48d-338421c5b92a None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:04.623884 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f32ddf6f-4f4b-4896-b48d-338421c5b92a None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:04.681108 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f98d5ad3-733b-4d84-b526-b01816eaa25f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:04.681729 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f98d5ad3-733b-4d84-b526-b01816eaa25f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:04.682006 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f98d5ad3-733b-4d84-b526-b01816eaa25f admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:04.682279 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f98d5ad3-733b-4d84-b526-b01816eaa25f admin admin] PATH_INFO: `/v3/projects/645b73a7f71d47c891fae7637fe36773` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:04.683398 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f98d5ad3-733b-4d84-b526-b01816eaa25f admin admin] RBAC: Authorizing `identity:delete_project(project_id=645b73a7f71d47c891fae7637fe36773)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:04.685359 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f98d5ad3-733b-4d84-b526-b01816eaa25f admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:04.729739 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-f98d5ad3-733b-4d84-b526-b01816eaa25f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '645b73a7f71d47c891fae7637fe36773', 'request_id': 'req-f98d5ad3-733b-4d84-b526-b01816eaa25f'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:04.735126 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-f98d5ad3-733b-4d84-b526-b01816eaa25f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '645b73a7f71d47c891fae7637fe36773', 'request_id': 'req-f98d5ad3-733b-4d84-b526-b01816eaa25f'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:04.741621 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-f98d5ad3-733b-4d84-b526-b01816eaa25f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '645b73a7f71d47c891fae7637fe36773', 'request_id': 'req-f98d5ad3-733b-4d84-b526-b01816eaa25f'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:04.754356 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-f98d5ad3-733b-4d84-b526-b01816eaa25f admin admin] The token cache is being invalidate because project 645b73a7f71d47c891fae7637fe36773 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 01:02:04.754642 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-f98d5ad3-733b-4d84-b526-b01816eaa25f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:04.755258 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-f98d5ad3-733b-4d84-b526-b01816eaa25f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:04.756746 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1286/2538] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 01:02:04 2022] DELETE /identity/v3/projects/645b73a7f71d47c891fae7637fe36773 => generated 0 bytes in 144 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:02:04.760868 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08e76568-7653-4659-8038-0c9151942c22 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:04.762355 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-08e76568-7653-4659-8038-0c9151942c22 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:04.775007 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08e76568-7653-4659-8038-0c9151942c22 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:04.798768 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=84,keystone:INSERT=6,keystone:DELETE=10 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:02:04.832789 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5898cff0-5993-49c2-a4d9-731cb69eaba5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:04.833424 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5898cff0-5993-49c2-a4d9-731cb69eaba5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:04.833732 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5898cff0-5993-49c2-a4d9-731cb69eaba5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:04.834008 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5898cff0-5993-49c2-a4d9-731cb69eaba5 admin admin] PATH_INFO: `/v3/users/1e2fc56fa35b49c6994cba0d4b949eee` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:04.835278 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5898cff0-5993-49c2-a4d9-731cb69eaba5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1e2fc56fa35b49c6994cba0d4b949eee)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:04.837133 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5898cff0-5993-49c2-a4d9-731cb69eaba5 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:04.881835 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-5898cff0-5993-49c2-a4d9-731cb69eaba5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1e2fc56fa35b49c6994cba0d4b949eee'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:04.889145 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-5898cff0-5993-49c2-a4d9-731cb69eaba5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1e2fc56fa35b49c6994cba0d4b949eee'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:04.889896 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-5898cff0-5993-49c2-a4d9-731cb69eaba5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1e2fc56fa35b49c6994cba0d4b949eee'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:04.896786 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-5898cff0-5993-49c2-a4d9-731cb69eaba5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1e2fc56fa35b49c6994cba0d4b949eee'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:04.908772 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-5898cff0-5993-49c2-a4d9-731cb69eaba5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1e2fc56fa35b49c6994cba0d4b949eee'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:04.914515 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1253/2539] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 01:02:04 2022] DELETE /identity/v3/users/1e2fc56fa35b49c6994cba0d4b949eee => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:02:05.086347 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f98d5ad3-733b-4d84-b526-b01816eaa25f admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:05.087847 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-f98d5ad3-733b-4d84-b526-b01816eaa25f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:05.100939 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f98d5ad3-733b-4d84-b526-b01816eaa25f admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:05.158430 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-362bd868-4f37-4fa4-8bec-e72e2f2fc394 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:05.159063 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-362bd868-4f37-4fa4-8bec-e72e2f2fc394 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:05.159341 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-362bd868-4f37-4fa4-8bec-e72e2f2fc394 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:05.160126 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-362bd868-4f37-4fa4-8bec-e72e2f2fc394 admin admin] PATH_INFO: `/v3/projects/77cd8ace7047445fb451ab960e27f19a` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:05.161241 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-362bd868-4f37-4fa4-8bec-e72e2f2fc394 admin admin] RBAC: Authorizing `identity:delete_project(project_id=77cd8ace7047445fb451ab960e27f19a)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:05.163132 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-362bd868-4f37-4fa4-8bec-e72e2f2fc394 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:05.187440 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:02:05.209944 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-362bd868-4f37-4fa4-8bec-e72e2f2fc394 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '77cd8ace7047445fb451ab960e27f19a', 'request_id': 'req-362bd868-4f37-4fa4-8bec-e72e2f2fc394'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:05.215369 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-362bd868-4f37-4fa4-8bec-e72e2f2fc394 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '77cd8ace7047445fb451ab960e27f19a', 'request_id': 'req-362bd868-4f37-4fa4-8bec-e72e2f2fc394'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:05.221292 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-362bd868-4f37-4fa4-8bec-e72e2f2fc394 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '77cd8ace7047445fb451ab960e27f19a', 'request_id': 'req-362bd868-4f37-4fa4-8bec-e72e2f2fc394'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:05.233851 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-362bd868-4f37-4fa4-8bec-e72e2f2fc394 admin admin] The token cache is being invalidate because project 77cd8ace7047445fb451ab960e27f19a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 01:02:05.234134 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-362bd868-4f37-4fa4-8bec-e72e2f2fc394 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:05.234744 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-362bd868-4f37-4fa4-8bec-e72e2f2fc394 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:05.236469 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1287/2540] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 01:02:05 2022] DELETE /identity/v3/projects/77cd8ace7047445fb451ab960e27f19a => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:02:05.241808 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bbd8c4c-cdeb-4207-882d-b3414cad4458 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:05.242098 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bbd8c4c-cdeb-4207-882d-b3414cad4458 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:05.242377 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bbd8c4c-cdeb-4207-882d-b3414cad4458 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:05.262204 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-6bbd8c4c-cdeb-4207-882d-b3414cad4458 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:02:05.263407 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-6bbd8c4c-cdeb-4207-882d-b3414cad4458 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:05.321442 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1254/2541] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:02:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:02:05.330318 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21255add-9b35-4805-8dcb-32f4616c8291 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:05.330609 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21255add-9b35-4805-8dcb-32f4616c8291 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:05.330884 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21255add-9b35-4805-8dcb-32f4616c8291 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:05.351426 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-21255add-9b35-4805-8dcb-32f4616c8291 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:02:05.352673 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-21255add-9b35-4805-8dcb-32f4616c8291 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:05.360476 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1288/2542] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:02:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:02:05.364779 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bbd8c4c-cdeb-4207-882d-b3414cad4458 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:05.374913 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bbd8c4c-cdeb-4207-882d-b3414cad4458 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:05.379721 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-643d0791-0486-49ac-ac6c-103d76e3e79e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:05.380340 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-643d0791-0486-49ac-ac6c-103d76e3e79e admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:05.380605 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-643d0791-0486-49ac-ac6c-103d76e3e79e admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:05.380879 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-643d0791-0486-49ac-ac6c-103d76e3e79e admin admin] PATH_INFO: `/v3/domains` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:05.381520 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-643d0791-0486-49ac-ac6c-103d76e3e79e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 01:02:05.381824 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-643d0791-0486-49ac-ac6c-103d76e3e79e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:05.383438 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-643d0791-0486-49ac-ac6c-103d76e3e79e admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:05.399441 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1255/2543] 10.222.0.33 () {58 vars in 1153 bytes} [Sun Oct 9 01:02:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 01:02:05.401052 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21255add-9b35-4805-8dcb-32f4616c8291 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:05.410926 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21255add-9b35-4805-8dcb-32f4616c8291 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:05.415569 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-013fcf21-3d6e-4ec1-b13d-587854d13a4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:05.416169 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-013fcf21-3d6e-4ec1-b13d-587854d13a4a admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:05.416448 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-013fcf21-3d6e-4ec1-b13d-587854d13a4a admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:05.416718 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-013fcf21-3d6e-4ec1-b13d-587854d13a4a admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:05.419453 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-013fcf21-3d6e-4ec1-b13d-587854d13a4a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:05.419453 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-013fcf21-3d6e-4ec1-b13d-587854d13a4a admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:05.450711 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1289/2544] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 01:02:05 2022] POST /identity/v3/projects => generated 363 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 01:02:05.451949 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-643d0791-0486-49ac-ac6c-103d76e3e79e admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:05.467849 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-643d0791-0486-49ac-ac6c-103d76e3e79e admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:05.503359 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:02:05.536341 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb17edcf-64dc-4de3-a024-55c14f820ee3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:05.536942 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb17edcf-64dc-4de3-a024-55c14f820ee3 admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:05.537222 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb17edcf-64dc-4de3-a024-55c14f820ee3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:05.537493 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb17edcf-64dc-4de3-a024-55c14f820ee3 admin admin] PATH_INFO: `/v3/users` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:05.538115 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb17edcf-64dc-4de3-a024-55c14f820ee3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:05.540079 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb17edcf-64dc-4de3-a024-55c14f820ee3 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:05.563634 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1256/2545] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 01:02:05 2022] POST /identity/v3/users => generated 282 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 01:02:05.566936 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-013fcf21-3d6e-4ec1-b13d-587854d13a4a admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:05.578582 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-013fcf21-3d6e-4ec1-b13d-587854d13a4a admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:05.584311 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0045d777-53ea-46a2-9ade-2c843202e3df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:05.584910 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0045d777-53ea-46a2-9ade-2c843202e3df admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:05.585189 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0045d777-53ea-46a2-9ade-2c843202e3df admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:05.585458 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0045d777-53ea-46a2-9ade-2c843202e3df admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:05.586089 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0045d777-53ea-46a2-9ade-2c843202e3df admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:05.587851 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0045d777-53ea-46a2-9ade-2c843202e3df admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:05.601999 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1290/2546] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 01:02:05 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 09 01:02:05.606046 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb17edcf-64dc-4de3-a024-55c14f820ee3 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:05.616010 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb17edcf-64dc-4de3-a024-55c14f820ee3 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:05.620827 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e03f0f5-6af7-446e-b89f-f803455c6d7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:05.621402 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e03f0f5-6af7-446e-b89f-f803455c6d7e admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:05.621686 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e03f0f5-6af7-446e-b89f-f803455c6d7e admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:05.621974 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e03f0f5-6af7-446e-b89f-f803455c6d7e admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:05.622627 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e03f0f5-6af7-446e-b89f-f803455c6d7e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:05.624235 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e03f0f5-6af7-446e-b89f-f803455c6d7e admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:05.635360 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1257/2547] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 01:02:05 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 09 01:02:05.640698 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0045d777-53ea-46a2-9ade-2c843202e3df admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:05.651516 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0045d777-53ea-46a2-9ade-2c843202e3df admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:05.656254 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8acc5b2-dff2-47ce-bfb6-9f32adb4d130 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:05.656927 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8acc5b2-dff2-47ce-bfb6-9f32adb4d130 admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:05.657208 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8acc5b2-dff2-47ce-bfb6-9f32adb4d130 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:05.657487 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8acc5b2-dff2-47ce-bfb6-9f32adb4d130 admin admin] PATH_INFO: `/v3/projects/53bd93d63bc743f881bae696798e2a0b/users/73b63e403f8746b1af6301956285e6a4/roles/588016933bab436d8b7c3ac76827356d` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:05.672653 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8acc5b2-dff2-47ce-bfb6-9f32adb4d130 admin admin] RBAC: Authorizing `identity:create_grant(project_id=53bd93d63bc743f881bae696798e2a0b, user_id=73b63e403f8746b1af6301956285e6a4, role_id=588016933bab436d8b7c3ac76827356d)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:05.675312 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8acc5b2-dff2-47ce-bfb6-9f32adb4d130 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:05.687557 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1291/2548] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 01:02:05 2022] PUT /identity/v3/projects/53bd93d63bc743f881bae696798e2a0b/users/73b63e403f8746b1af6301956285e6a4/roles/588016933bab436d8b7c3ac76827356d => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:02:05.698011 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-110f0132-6006-4374-a018-6dbb0bd0aab3 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:05.698306 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-110f0132-6006-4374-a018-6dbb0bd0aab3 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:05.698601 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-110f0132-6006-4374-a018-6dbb0bd0aab3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:05.728007 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-110f0132-6006-4374-a018-6dbb0bd0aab3 None None] MFA Rules not processed for user `73b63e403f8746b1af6301956285e6a4`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:02:05.729161 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-110f0132-6006-4374-a018-6dbb0bd0aab3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:05.791879 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1258/2549] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:02:05 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:02:05.796446 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8acc5b2-dff2-47ce-bfb6-9f32adb4d130 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:05.857829 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8acc5b2-dff2-47ce-bfb6-9f32adb4d130 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:05.862570 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22e061b0-452a-4e83-bdfd-024bbe1d9247 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:05.863168 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22e061b0-452a-4e83-bdfd-024bbe1d9247 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:05.863450 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22e061b0-452a-4e83-bdfd-024bbe1d9247 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:05.863765 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22e061b0-452a-4e83-bdfd-024bbe1d9247 admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:05.864404 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22e061b0-452a-4e83-bdfd-024bbe1d9247 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:05.866305 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22e061b0-452a-4e83-bdfd-024bbe1d9247 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:05.894899 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1292/2550] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 01:02:05 2022] POST /identity/v3/projects => generated 363 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 01:02:05.899254 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-110f0132-6006-4374-a018-6dbb0bd0aab3 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:05.910208 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-110f0132-6006-4374-a018-6dbb0bd0aab3 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:05.969448 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6c83310-befc-43fb-9b99-c13ad08d31c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:05.970053 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6c83310-befc-43fb-9b99-c13ad08d31c7 admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:05.970328 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6c83310-befc-43fb-9b99-c13ad08d31c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:05.970596 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6c83310-befc-43fb-9b99-c13ad08d31c7 admin admin] PATH_INFO: `/v3/users` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:05.971207 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6c83310-befc-43fb-9b99-c13ad08d31c7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:05.973171 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6c83310-befc-43fb-9b99-c13ad08d31c7 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:05.993383 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1259/2551] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 01:02:05 2022] POST /identity/v3/users => generated 288 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 01:02:05.997349 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22e061b0-452a-4e83-bdfd-024bbe1d9247 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:06.008151 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22e061b0-452a-4e83-bdfd-024bbe1d9247 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:06.012719 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4ea74c2-0bb7-4803-a9ce-3fcd43e46bbe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:06.013309 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ea74c2-0bb7-4803-a9ce-3fcd43e46bbe admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:06.013590 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ea74c2-0bb7-4803-a9ce-3fcd43e46bbe admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:06.013867 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ea74c2-0bb7-4803-a9ce-3fcd43e46bbe admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:06.014502 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4ea74c2-0bb7-4803-a9ce-3fcd43e46bbe admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:06.016149 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4ea74c2-0bb7-4803-a9ce-3fcd43e46bbe admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:06.027433 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1293/2552] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 01:02:05 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 01:02:06.031385 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6c83310-befc-43fb-9b99-c13ad08d31c7 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:06.041140 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6c83310-befc-43fb-9b99-c13ad08d31c7 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:06.045632 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e790414f-8d92-4fb2-9ad7-9bb433af6322 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:06.046257 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e790414f-8d92-4fb2-9ad7-9bb433af6322 admin admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:06.046534 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e790414f-8d92-4fb2-9ad7-9bb433af6322 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:06.046806 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e790414f-8d92-4fb2-9ad7-9bb433af6322 admin admin] PATH_INFO: `/v3/projects/d7ab625af44d4b25894a9bffbbddfa4c/users/0bdd5405edbf42fe9927e235def045c8/roles/092036ee46ac427baf4d56468575f697` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:06.061038 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e790414f-8d92-4fb2-9ad7-9bb433af6322 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d7ab625af44d4b25894a9bffbbddfa4c, user_id=0bdd5405edbf42fe9927e235def045c8, role_id=092036ee46ac427baf4d56468575f697)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:06.063687 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e790414f-8d92-4fb2-9ad7-9bb433af6322 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:06.077090 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1260/2553] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 01:02:06 2022] PUT /identity/v3/projects/d7ab625af44d4b25894a9bffbbddfa4c/users/0bdd5405edbf42fe9927e235def045c8/roles/092036ee46ac427baf4d56468575f697 => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:02:06.080820 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4ea74c2-0bb7-4803-a9ce-3fcd43e46bbe admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:06.091322 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4ea74c2-0bb7-4803-a9ce-3fcd43e46bbe admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:06.109844 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=90,keystone:DELETE=6,keystone:INSERT=4 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:02:06.149303 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00b6a1a1-87a4-4869-a353-fbdb8d4066c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:06.149898 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00b6a1a1-87a4-4869-a353-fbdb8d4066c0 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:06.150177 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00b6a1a1-87a4-4869-a353-fbdb8d4066c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:06.150449 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00b6a1a1-87a4-4869-a353-fbdb8d4066c0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:06.151343 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00b6a1a1-87a4-4869-a353-fbdb8d4066c0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:06.153256 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00b6a1a1-87a4-4869-a353-fbdb8d4066c0 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:06.163257 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1294/2554] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 01:02:06 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 01:02:06.167376 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e790414f-8d92-4fb2-9ad7-9bb433af6322 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:06.178115 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e790414f-8d92-4fb2-9ad7-9bb433af6322 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:06.182768 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c21edbb6-b276-4fc5-912d-ea6533070ebb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:06.183404 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c21edbb6-b276-4fc5-912d-ea6533070ebb admin admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:06.183715 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c21edbb6-b276-4fc5-912d-ea6533070ebb admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:06.183992 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c21edbb6-b276-4fc5-912d-ea6533070ebb admin admin] PATH_INFO: `/v3/projects/d7ab625af44d4b25894a9bffbbddfa4c/users/0bdd5405edbf42fe9927e235def045c8/roles/092036ee46ac427baf4d56468575f697` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:06.185878 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c21edbb6-b276-4fc5-912d-ea6533070ebb admin admin] RBAC: Authorizing `identity:create_grant(project_id=d7ab625af44d4b25894a9bffbbddfa4c, user_id=0bdd5405edbf42fe9927e235def045c8, role_id=092036ee46ac427baf4d56468575f697)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:06.188573 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c21edbb6-b276-4fc5-912d-ea6533070ebb admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:06.190728 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:02:06.200645 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1261/2555] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 01:02:06 2022] PUT /identity/v3/projects/d7ab625af44d4b25894a9bffbbddfa4c/users/0bdd5405edbf42fe9927e235def045c8/roles/092036ee46ac427baf4d56468575f697 => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:02:06.210877 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-434d5a04-1ca5-4810-8e8d-5d202f3d160e None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:06.211163 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-434d5a04-1ca5-4810-8e8d-5d202f3d160e None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:06.211433 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-434d5a04-1ca5-4810-8e8d-5d202f3d160e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:06.238613 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-434d5a04-1ca5-4810-8e8d-5d202f3d160e None None] MFA Rules not processed for user `0bdd5405edbf42fe9927e235def045c8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:02:06.239805 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-434d5a04-1ca5-4810-8e8d-5d202f3d160e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:06.305256 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1295/2556] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:02:06 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:02:06.316943 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c21edbb6-b276-4fc5-912d-ea6533070ebb admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:06.318659 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-c21edbb6-b276-4fc5-912d-ea6533070ebb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:06.375297 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c21edbb6-b276-4fc5-912d-ea6533070ebb admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:06.379572 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80c6a8d2-9578-43e5-8c19-4ec72065cb06 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3f48335668a4e8f934ac11ff66cd308', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:06.380196 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c6a8d2-9578-43e5-8c19-4ec72065cb06 service cinder] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:06.380475 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c6a8d2-9578-43e5-8c19-4ec72065cb06 service cinder] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:06.380746 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c6a8d2-9578-43e5-8c19-4ec72065cb06 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:06.389650 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80c6a8d2-9578-43e5-8c19-4ec72065cb06 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:06.391103 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80c6a8d2-9578-43e5-8c19-4ec72065cb06 service cinder] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:06.400576 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1262/2557] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 01:02:06 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 01:02:06.475711 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-434d5a04-1ca5-4810-8e8d-5d202f3d160e None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:06.485085 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-434d5a04-1ca5-4810-8e8d-5d202f3d160e None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:06.489269 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c669e78-5e3d-4194-a8d5-9f6a493944a9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3f48335668a4e8f934ac11ff66cd308', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:06.489847 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c669e78-5e3d-4194-a8d5-9f6a493944a9 service cinder] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:06.490120 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c669e78-5e3d-4194-a8d5-9f6a493944a9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:06.490386 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c669e78-5e3d-4194-a8d5-9f6a493944a9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:06.544703 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c669e78-5e3d-4194-a8d5-9f6a493944a9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:06.546119 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c669e78-5e3d-4194-a8d5-9f6a493944a9 service cinder] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:06.555348 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1296/2558] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 01:02:06 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 80 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 01:02:06.751754 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80c6a8d2-9578-43e5-8c19-4ec72065cb06 service cinder] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:06.753186 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-80c6a8d2-9578-43e5-8c19-4ec72065cb06 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:06.808909 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80c6a8d2-9578-43e5-8c19-4ec72065cb06 service cinder] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:06.812412 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ce6a094-9960-43fd-8b54-e86458129065 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:06.813008 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce6a094-9960-43fd-8b54-e86458129065 service neutron] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:06.813288 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce6a094-9960-43fd-8b54-e86458129065 service neutron] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:06.813562 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce6a094-9960-43fd-8b54-e86458129065 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:06.815227 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-5ce6a094-9960-43fd-8b54-e86458129065 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:06.877099 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ce6a094-9960-43fd-8b54-e86458129065 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:06.878595 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ce6a094-9960-43fd-8b54-e86458129065 service neutron] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:06.889503 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1263/2559] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 01:02:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 138 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 01:02:09.319915 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c669e78-5e3d-4194-a8d5-9f6a493944a9 service cinder] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:09.330834 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c669e78-5e3d-4194-a8d5-9f6a493944a9 service cinder] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:09.335783 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7de0579b-26a0-465d-ae4a-4c58199f66c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:09.336349 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7de0579b-26a0-465d-ae4a-4c58199f66c4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:09.336638 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7de0579b-26a0-465d-ae4a-4c58199f66c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:09.336891 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7de0579b-26a0-465d-ae4a-4c58199f66c4 admin admin] PATH_INFO: `/v3/users/73b63e403f8746b1af6301956285e6a4` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:09.337994 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7de0579b-26a0-465d-ae4a-4c58199f66c4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=73b63e403f8746b1af6301956285e6a4)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:09.340076 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7de0579b-26a0-465d-ae4a-4c58199f66c4 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:09.387527 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:02:09.399917 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-7de0579b-26a0-465d-ae4a-4c58199f66c4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '73b63e403f8746b1af6301956285e6a4'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:09.408306 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-7de0579b-26a0-465d-ae4a-4c58199f66c4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '73b63e403f8746b1af6301956285e6a4'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:09.409076 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-7de0579b-26a0-465d-ae4a-4c58199f66c4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '73b63e403f8746b1af6301956285e6a4'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:09.409735 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-7de0579b-26a0-465d-ae4a-4c58199f66c4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '73b63e403f8746b1af6301956285e6a4'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:09.419807 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-7de0579b-26a0-465d-ae4a-4c58199f66c4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '73b63e403f8746b1af6301956285e6a4'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:09.444447 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1297/2560] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 01:02:09 2022] DELETE /identity/v3/users/73b63e403f8746b1af6301956285e6a4 => generated 0 bytes in 125 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 09 01:02:09.449226 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae8137f4-a266-4eed-9240-68c12893674d None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:09.449715 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae8137f4-a266-4eed-9240-68c12893674d None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:09.449833 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae8137f4-a266-4eed-9240-68c12893674d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:09.481740 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-ae8137f4-a266-4eed-9240-68c12893674d None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:02:09.482979 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-ae8137f4-a266-4eed-9240-68c12893674d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:09.528241 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:02:09.550825 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1264/2561] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:02:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 09 01:02:09.560755 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7de0579b-26a0-465d-ae4a-4c58199f66c4 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:09.562250 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-7de0579b-26a0-465d-ae4a-4c58199f66c4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:09.627460 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7de0579b-26a0-465d-ae4a-4c58199f66c4 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:09.631182 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db61e2a7-27be-4af5-93ff-3a69d53db043 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:09.631818 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db61e2a7-27be-4af5-93ff-3a69d53db043 service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:09.632104 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db61e2a7-27be-4af5-93ff-3a69d53db043 service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:09.632377 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db61e2a7-27be-4af5-93ff-3a69d53db043 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:09.641746 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db61e2a7-27be-4af5-93ff-3a69d53db043 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:09.643211 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db61e2a7-27be-4af5-93ff-3a69d53db043 service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:09.652935 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1298/2562] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 01:02:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 01:02:09.949273 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae8137f4-a266-4eed-9240-68c12893674d None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:09.950736 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-ae8137f4-a266-4eed-9240-68c12893674d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:09.968952 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae8137f4-a266-4eed-9240-68c12893674d None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:09.974965 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9827bf8-8a11-4971-aa51-bc429fa4c9ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:09.975878 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9827bf8-8a11-4971-aa51-bc429fa4c9ad admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:09.976463 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9827bf8-8a11-4971-aa51-bc429fa4c9ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:09.976643 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9827bf8-8a11-4971-aa51-bc429fa4c9ad admin admin] PATH_INFO: `/v3/projects/53bd93d63bc743f881bae696798e2a0b` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:09.977843 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9827bf8-8a11-4971-aa51-bc429fa4c9ad admin admin] RBAC: Authorizing `identity:delete_project(project_id=53bd93d63bc743f881bae696798e2a0b)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:09.981854 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9827bf8-8a11-4971-aa51-bc429fa4c9ad admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:10.043822 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-a9827bf8-8a11-4971-aa51-bc429fa4c9ad admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '53bd93d63bc743f881bae696798e2a0b', 'request_id': 'req-a9827bf8-8a11-4971-aa51-bc429fa4c9ad'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:10.050265 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-a9827bf8-8a11-4971-aa51-bc429fa4c9ad admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '53bd93d63bc743f881bae696798e2a0b', 'request_id': 'req-a9827bf8-8a11-4971-aa51-bc429fa4c9ad'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:10.057223 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-a9827bf8-8a11-4971-aa51-bc429fa4c9ad admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '53bd93d63bc743f881bae696798e2a0b', 'request_id': 'req-a9827bf8-8a11-4971-aa51-bc429fa4c9ad'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:10.071877 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-a9827bf8-8a11-4971-aa51-bc429fa4c9ad admin admin] The token cache is being invalidate because project 53bd93d63bc743f881bae696798e2a0b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81453) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 01:02:10.072095 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-a9827bf8-8a11-4971-aa51-bc429fa4c9ad admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:10.077249 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-a9827bf8-8a11-4971-aa51-bc429fa4c9ad admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:10.078788 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1265/2563] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 01:02:09 2022] DELETE /identity/v3/projects/53bd93d63bc743f881bae696798e2a0b => generated 0 bytes in 130 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:02:10.085726 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db61e2a7-27be-4af5-93ff-3a69d53db043 service neutron] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:10.098979 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9827bf8-8a11-4971-aa51-bc429fa4c9ad admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:10.099509 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db61e2a7-27be-4af5-93ff-3a69d53db043 service neutron] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:10.100532 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-a9827bf8-8a11-4971-aa51-bc429fa4c9ad admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:10.172103 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a37a7a3e-e5ae-43f9-ad2e-f2c8c6e7fa3f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:10.175639 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a37a7a3e-e5ae-43f9-ad2e-f2c8c6e7fa3f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:10.175639 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a37a7a3e-e5ae-43f9-ad2e-f2c8c6e7fa3f admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:10.175639 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a37a7a3e-e5ae-43f9-ad2e-f2c8c6e7fa3f admin admin] PATH_INFO: `/v3/users/0bdd5405edbf42fe9927e235def045c8` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:10.175639 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a37a7a3e-e5ae-43f9-ad2e-f2c8c6e7fa3f admin admin] RBAC: Authorizing `identity:delete_user(user_id=0bdd5405edbf42fe9927e235def045c8)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:10.176529 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a37a7a3e-e5ae-43f9-ad2e-f2c8c6e7fa3f admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:10.181202 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9827bf8-8a11-4971-aa51-bc429fa4c9ad admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:10.186482 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80247ad1-ce93-4bba-a0c2-a7f9599d29e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:10.187136 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80247ad1-ce93-4bba-a0c2-a7f9599d29e0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:10.187545 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80247ad1-ce93-4bba-a0c2-a7f9599d29e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:10.187839 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80247ad1-ce93-4bba-a0c2-a7f9599d29e0 admin admin] PATH_INFO: `/v3/users/b027fb8adadc488e975c4541e7ee6626` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:10.188961 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80247ad1-ce93-4bba-a0c2-a7f9599d29e0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b027fb8adadc488e975c4541e7ee6626)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:10.190911 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80247ad1-ce93-4bba-a0c2-a7f9599d29e0 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:10.227432 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-a37a7a3e-e5ae-43f9-ad2e-f2c8c6e7fa3f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0bdd5405edbf42fe9927e235def045c8'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:10.234673 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-a37a7a3e-e5ae-43f9-ad2e-f2c8c6e7fa3f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0bdd5405edbf42fe9927e235def045c8'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:10.235351 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-a37a7a3e-e5ae-43f9-ad2e-f2c8c6e7fa3f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0bdd5405edbf42fe9927e235def045c8'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:10.235972 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-a37a7a3e-e5ae-43f9-ad2e-f2c8c6e7fa3f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0bdd5405edbf42fe9927e235def045c8'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:10.238564 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:02:10.240456 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-80247ad1-ce93-4bba-a0c2-a7f9599d29e0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b027fb8adadc488e975c4541e7ee6626'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:10.244464 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-a37a7a3e-e5ae-43f9-ad2e-f2c8c6e7fa3f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0bdd5405edbf42fe9927e235def045c8'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:10.251224 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-80247ad1-ce93-4bba-a0c2-a7f9599d29e0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b027fb8adadc488e975c4541e7ee6626'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:10.251993 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-80247ad1-ce93-4bba-a0c2-a7f9599d29e0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b027fb8adadc488e975c4541e7ee6626'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:10.260397 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-80247ad1-ce93-4bba-a0c2-a7f9599d29e0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b027fb8adadc488e975c4541e7ee6626'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:10.265120 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1299/2564] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 01:02:10 2022] DELETE /identity/v3/users/0bdd5405edbf42fe9927e235def045c8 => generated 0 bytes in 182 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:02:10.268705 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=82,keystone:DELETE=16,keystone:INSERT=2 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:02:10.276031 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-80247ad1-ce93-4bba-a0c2-a7f9599d29e0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b027fb8adadc488e975c4541e7ee6626'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:10.283375 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1266/2565] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 01:02:10 2022] DELETE /identity/v3/users/b027fb8adadc488e975c4541e7ee6626 => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 09 01:02:10.527854 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a37a7a3e-e5ae-43f9-ad2e-f2c8c6e7fa3f admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:10.529816 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-a37a7a3e-e5ae-43f9-ad2e-f2c8c6e7fa3f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:10.607841 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a37a7a3e-e5ae-43f9-ad2e-f2c8c6e7fa3f admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:10.608005 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80247ad1-ce93-4bba-a0c2-a7f9599d29e0 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:10.609381 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-80247ad1-ce93-4bba-a0c2-a7f9599d29e0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:10.674516 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d306b24b-536b-45ea-9991-ca5613b361e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:10.675694 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d306b24b-536b-45ea-9991-ca5613b361e1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:10.675976 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d306b24b-536b-45ea-9991-ca5613b361e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:10.676251 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d306b24b-536b-45ea-9991-ca5613b361e1 admin admin] PATH_INFO: `/v3/projects/a7d8e82f58814da6b8c08bfa04955d4c` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:10.677348 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d306b24b-536b-45ea-9991-ca5613b361e1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a7d8e82f58814da6b8c08bfa04955d4c)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:10.677795 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80247ad1-ce93-4bba-a0c2-a7f9599d29e0 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:10.679240 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d306b24b-536b-45ea-9991-ca5613b361e1 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:10.682848 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03071d8c-1e32-430f-bffe-479a04d4d8ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:10.683479 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03071d8c-1e32-430f-bffe-479a04d4d8ce admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:10.683792 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03071d8c-1e32-430f-bffe-479a04d4d8ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:10.684070 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03071d8c-1e32-430f-bffe-479a04d4d8ce admin admin] PATH_INFO: `/v3/projects/d7ab625af44d4b25894a9bffbbddfa4c` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:10.685255 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03071d8c-1e32-430f-bffe-479a04d4d8ce admin admin] RBAC: Authorizing `identity:delete_project(project_id=d7ab625af44d4b25894a9bffbbddfa4c)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:10.687130 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03071d8c-1e32-430f-bffe-479a04d4d8ce admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:10.725375 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-d306b24b-536b-45ea-9991-ca5613b361e1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a7d8e82f58814da6b8c08bfa04955d4c', 'request_id': 'req-d306b24b-536b-45ea-9991-ca5613b361e1'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:10.731126 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-d306b24b-536b-45ea-9991-ca5613b361e1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a7d8e82f58814da6b8c08bfa04955d4c', 'request_id': 'req-d306b24b-536b-45ea-9991-ca5613b361e1'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:10.735292 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-03071d8c-1e32-430f-bffe-479a04d4d8ce admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd7ab625af44d4b25894a9bffbbddfa4c', 'request_id': 'req-03071d8c-1e32-430f-bffe-479a04d4d8ce'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:10.739033 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-d306b24b-536b-45ea-9991-ca5613b361e1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a7d8e82f58814da6b8c08bfa04955d4c', 'request_id': 'req-d306b24b-536b-45ea-9991-ca5613b361e1'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:10.741993 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-03071d8c-1e32-430f-bffe-479a04d4d8ce admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd7ab625af44d4b25894a9bffbbddfa4c', 'request_id': 'req-03071d8c-1e32-430f-bffe-479a04d4d8ce'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:10.748721 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-03071d8c-1e32-430f-bffe-479a04d4d8ce admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd7ab625af44d4b25894a9bffbbddfa4c', 'request_id': 'req-03071d8c-1e32-430f-bffe-479a04d4d8ce'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:10.751787 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-d306b24b-536b-45ea-9991-ca5613b361e1 admin admin] The token cache is being invalidate because project a7d8e82f58814da6b8c08bfa04955d4c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 01:02:10.751996 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-d306b24b-536b-45ea-9991-ca5613b361e1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:10.752752 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-d306b24b-536b-45ea-9991-ca5613b361e1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:10.754993 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1300/2566] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 01:02:10 2022] DELETE /identity/v3/projects/a7d8e82f58814da6b8c08bfa04955d4c => generated 0 bytes in 228 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:02:10.759952 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84fa5cea-f612-4955-ab00-beda55c1a2c2 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:10.760239 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84fa5cea-f612-4955-ab00-beda55c1a2c2 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:10.760513 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84fa5cea-f612-4955-ab00-beda55c1a2c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:10.761577 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-03071d8c-1e32-430f-bffe-479a04d4d8ce admin admin] The token cache is being invalidate because project d7ab625af44d4b25894a9bffbbddfa4c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81453) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 01:02:10.761870 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-03071d8c-1e32-430f-bffe-479a04d4d8ce admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:10.762514 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-03071d8c-1e32-430f-bffe-479a04d4d8ce admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:10.764203 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1267/2567] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 01:02:10 2022] DELETE /identity/v3/projects/d7ab625af44d4b25894a9bffbbddfa4c => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 09 01:02:10.768543 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1549af5a-2f40-4989-9fcd-ac4f4ad95816 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:10.768826 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1549af5a-2f40-4989-9fcd-ac4f4ad95816 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:10.769130 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1549af5a-2f40-4989-9fcd-ac4f4ad95816 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:10.780730 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-84fa5cea-f612-4955-ab00-beda55c1a2c2 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:02:10.781940 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-84fa5cea-f612-4955-ab00-beda55c1a2c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:10.789478 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-1549af5a-2f40-4989-9fcd-ac4f4ad95816 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:02:10.790586 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:02:10.790751 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-1549af5a-2f40-4989-9fcd-ac4f4ad95816 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:10.849371 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1301/2568] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:02:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:02:10.851921 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1268/2569] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:02:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 09 01:02:10.857245 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6b312b4-e2bc-48fb-b9ff-939018a9f1c3 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:10.857539 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6b312b4-e2bc-48fb-b9ff-939018a9f1c3 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:10.857815 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6b312b4-e2bc-48fb-b9ff-939018a9f1c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:10.859759 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe1b273-b757-4c1a-b4da-74980fc7c4a8 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:10.860220 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe1b273-b757-4c1a-b4da-74980fc7c4a8 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:10.860756 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe1b273-b757-4c1a-b4da-74980fc7c4a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:10.879086 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-b6b312b4-e2bc-48fb-b9ff-939018a9f1c3 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:02:10.880338 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-b6b312b4-e2bc-48fb-b9ff-939018a9f1c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:10.881644 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-abe1b273-b757-4c1a-b4da-74980fc7c4a8 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:02:10.882936 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-abe1b273-b757-4c1a-b4da-74980fc7c4a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:10.891838 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1302/2570] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:02:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:02:10.894009 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6b312b4-e2bc-48fb-b9ff-939018a9f1c3 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:10.895679 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1269/2571] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:02:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:02:10.896617 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abe1b273-b757-4c1a-b4da-74980fc7c4a8 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:10.905295 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6b312b4-e2bc-48fb-b9ff-939018a9f1c3 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:10.906045 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abe1b273-b757-4c1a-b4da-74980fc7c4a8 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:10.910078 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-665680dc-3b66-473a-b872-735e010c08d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:10.910659 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96362f14-9273-48f8-83e1-27a73256887d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:10.910816 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-665680dc-3b66-473a-b872-735e010c08d2 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:10.910970 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-665680dc-3b66-473a-b872-735e010c08d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:10.911216 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-665680dc-3b66-473a-b872-735e010c08d2 admin admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:10.911326 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96362f14-9273-48f8-83e1-27a73256887d admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:10.911527 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96362f14-9273-48f8-83e1-27a73256887d admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:10.911837 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96362f14-9273-48f8-83e1-27a73256887d admin admin] PATH_INFO: `/v3/domains` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:10.911947 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-665680dc-3b66-473a-b872-735e010c08d2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 01:02:10.912178 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-665680dc-3b66-473a-b872-735e010c08d2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:10.912480 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96362f14-9273-48f8-83e1-27a73256887d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 01:02:10.912785 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96362f14-9273-48f8-83e1-27a73256887d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:10.913815 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-665680dc-3b66-473a-b872-735e010c08d2 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:10.914432 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96362f14-9273-48f8-83e1-27a73256887d admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:10.925052 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:02:10.927645 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1303/2572] 10.222.0.33 () {58 vars in 1153 bytes} [Sun Oct 9 01:02:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 01:02:10.931612 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1270/2573] 10.222.0.33 () {58 vars in 1153 bytes} [Sun Oct 9 01:02:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 01:02:10.932767 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-665680dc-3b66-473a-b872-735e010c08d2 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:10.934596 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96362f14-9273-48f8-83e1-27a73256887d admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:10.947071 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96362f14-9273-48f8-83e1-27a73256887d admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:10.948637 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-665680dc-3b66-473a-b872-735e010c08d2 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:10.951750 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c34613-c14e-4bc3-b534-c08e8cb0e746 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:10.952465 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c34613-c14e-4bc3-b534-c08e8cb0e746 admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:10.952723 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c34613-c14e-4bc3-b534-c08e8cb0e746 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:10.952914 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c34613-c14e-4bc3-b534-c08e8cb0e746 admin admin] PATH_INFO: `/v3/projects` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:10.953248 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3917f516-73a8-4b52-868f-a10769c5eae8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:10.953506 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0c34613-c14e-4bc3-b534-c08e8cb0e746 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:10.953840 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3917f516-73a8-4b52-868f-a10769c5eae8 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:10.954154 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3917f516-73a8-4b52-868f-a10769c5eae8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:10.954423 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3917f516-73a8-4b52-868f-a10769c5eae8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:10.955006 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3917f516-73a8-4b52-868f-a10769c5eae8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:10.955383 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0c34613-c14e-4bc3-b534-c08e8cb0e746 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:10.958198 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3917f516-73a8-4b52-868f-a10769c5eae8 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:10.987440 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1271/2574] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 01:02:10 2022] POST /identity/v3/projects => generated 369 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 01:02:10.991963 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c34613-c14e-4bc3-b534-c08e8cb0e746 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:10.992842 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1304/2575] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 01:02:10 2022] POST /identity/v3/projects => generated 371 bytes in 60 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 01:02:10.996565 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3917f516-73a8-4b52-868f-a10769c5eae8 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:11.055510 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c34613-c14e-4bc3-b534-c08e8cb0e746 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:11.059693 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3917f516-73a8-4b52-868f-a10769c5eae8 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:11.060224 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-219c2464-e5df-4270-baea-8ee18aa2c22e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:11.060824 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-219c2464-e5df-4270-baea-8ee18aa2c22e admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:11.061107 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-219c2464-e5df-4270-baea-8ee18aa2c22e admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:11.061431 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-219c2464-e5df-4270-baea-8ee18aa2c22e admin admin] PATH_INFO: `/v3/users` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:11.061988 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-219c2464-e5df-4270-baea-8ee18aa2c22e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:11.063905 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-219c2464-e5df-4270-baea-8ee18aa2c22e admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:11.064374 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05d815b0-f42a-4e51-a25f-5a703a0fcb71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:11.064971 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d815b0-f42a-4e51-a25f-5a703a0fcb71 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:11.065254 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d815b0-f42a-4e51-a25f-5a703a0fcb71 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:11.065530 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d815b0-f42a-4e51-a25f-5a703a0fcb71 admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:11.066230 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05d815b0-f42a-4e51-a25f-5a703a0fcb71 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:11.068123 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05d815b0-f42a-4e51-a25f-5a703a0fcb71 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:11.088651 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1272/2576] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 01:02:10 2022] POST /identity/v3/users => generated 285 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 09 01:02:11.092231 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1305/2577] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 01:02:10 2022] POST /identity/v3/users => generated 286 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 09 01:02:11.092741 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-219c2464-e5df-4270-baea-8ee18aa2c22e admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:11.095907 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05d815b0-f42a-4e51-a25f-5a703a0fcb71 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:11.104864 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-219c2464-e5df-4270-baea-8ee18aa2c22e admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:11.105999 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05d815b0-f42a-4e51-a25f-5a703a0fcb71 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:11.109463 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a7897fa-a620-4fc3-b09b-7ec766acea87 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:11.110083 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7897fa-a620-4fc3-b09b-7ec766acea87 admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:11.110322 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7897fa-a620-4fc3-b09b-7ec766acea87 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:11.110596 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7897fa-a620-4fc3-b09b-7ec766acea87 admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:11.110712 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6356b3c-d3fd-4043-89af-141b343bd415 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:11.111175 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6356b3c-d3fd-4043-89af-141b343bd415 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:11.111286 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a7897fa-a620-4fc3-b09b-7ec766acea87 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:11.111457 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6356b3c-d3fd-4043-89af-141b343bd415 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:11.111766 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6356b3c-d3fd-4043-89af-141b343bd415 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:11.112399 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6356b3c-d3fd-4043-89af-141b343bd415 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:11.113526 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a7897fa-a620-4fc3-b09b-7ec766acea87 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:11.114536 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6356b3c-d3fd-4043-89af-141b343bd415 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:11.123702 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1273/2578] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 01:02:11 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 01:02:11.126703 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1306/2579] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 01:02:11 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 09 01:02:11.127724 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a7897fa-a620-4fc3-b09b-7ec766acea87 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:11.127976 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6356b3c-d3fd-4043-89af-141b343bd415 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:11.137782 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6356b3c-d3fd-4043-89af-141b343bd415 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:11.139645 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a7897fa-a620-4fc3-b09b-7ec766acea87 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:11.142524 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8141637-a52b-4b0e-b072-8f4531de0f5b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:11.143055 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8141637-a52b-4b0e-b072-8f4531de0f5b admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:11.143335 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8141637-a52b-4b0e-b072-8f4531de0f5b admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:11.143634 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8141637-a52b-4b0e-b072-8f4531de0f5b admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:11.144322 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8141637-a52b-4b0e-b072-8f4531de0f5b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:11.144547 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-193c44bc-a678-42a8-b67a-a1c0c7acb454 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:11.145141 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-193c44bc-a678-42a8-b67a-a1c0c7acb454 admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:11.145423 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-193c44bc-a678-42a8-b67a-a1c0c7acb454 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:11.145696 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-193c44bc-a678-42a8-b67a-a1c0c7acb454 admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:11.145902 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8141637-a52b-4b0e-b072-8f4531de0f5b admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:11.146317 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-193c44bc-a678-42a8-b67a-a1c0c7acb454 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:11.148255 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-193c44bc-a678-42a8-b67a-a1c0c7acb454 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:11.156899 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1307/2580] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 01:02:11 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 01:02:11.161579 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8141637-a52b-4b0e-b072-8f4531de0f5b admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:11.167154 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1274/2581] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 01:02:11 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 09 01:02:11.168157 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-193c44bc-a678-42a8-b67a-a1c0c7acb454 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:11.174204 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8141637-a52b-4b0e-b072-8f4531de0f5b admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:11.179901 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-193c44bc-a678-42a8-b67a-a1c0c7acb454 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:11.180117 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9c5f124-3204-4918-a996-89a14d51f40c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:11.180670 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c5f124-3204-4918-a996-89a14d51f40c admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:11.180951 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c5f124-3204-4918-a996-89a14d51f40c admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:11.181220 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c5f124-3204-4918-a996-89a14d51f40c admin admin] PATH_INFO: `/v3/projects/97b9086fac714440938c6c50bd5026c3/users/05e4d867a47c4c6bbfeeb98041cc1dc1/roles/588016933bab436d8b7c3ac76827356d` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:11.186448 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64c625e3-d57c-4f97-9015-4db3dc693500 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:11.187133 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c625e3-d57c-4f97-9015-4db3dc693500 admin admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:11.187461 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c625e3-d57c-4f97-9015-4db3dc693500 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:11.187826 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c625e3-d57c-4f97-9015-4db3dc693500 admin admin] PATH_INFO: `/v3/projects/d9b18dc3d0f14cb982a654cba1c40532/users/d567661eb2c74eef9d7ef12176929037/roles/588016933bab436d8b7c3ac76827356d` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:11.196259 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9c5f124-3204-4918-a996-89a14d51f40c admin admin] RBAC: Authorizing `identity:create_grant(project_id=97b9086fac714440938c6c50bd5026c3, user_id=05e4d867a47c4c6bbfeeb98041cc1dc1, role_id=588016933bab436d8b7c3ac76827356d)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:11.198906 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9c5f124-3204-4918-a996-89a14d51f40c admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:11.208902 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64c625e3-d57c-4f97-9015-4db3dc693500 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d9b18dc3d0f14cb982a654cba1c40532, user_id=d567661eb2c74eef9d7ef12176929037, role_id=588016933bab436d8b7c3ac76827356d)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:11.212176 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64c625e3-d57c-4f97-9015-4db3dc693500 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:11.215716 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1308/2582] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 01:02:11 2022] PUT /identity/v3/projects/97b9086fac714440938c6c50bd5026c3/users/05e4d867a47c4c6bbfeeb98041cc1dc1/roles/588016933bab436d8b7c3ac76827356d => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 09 01:02:11.226074 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1275/2583] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 01:02:11 2022] PUT /identity/v3/projects/d9b18dc3d0f14cb982a654cba1c40532/users/d567661eb2c74eef9d7ef12176929037/roles/588016933bab436d8b7c3ac76827356d => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 09 01:02:11.226371 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf13a41-f935-44c7-895c-25a875cdc6f5 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:11.226766 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf13a41-f935-44c7-895c-25a875cdc6f5 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:11.227098 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf13a41-f935-44c7-895c-25a875cdc6f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:11.236074 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b1b337b-f588-4a05-a011-8ae13505387d None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:11.236468 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b1b337b-f588-4a05-a011-8ae13505387d None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:11.236743 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b1b337b-f588-4a05-a011-8ae13505387d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:11.255445 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-ecf13a41-f935-44c7-895c-25a875cdc6f5 None None] MFA Rules not processed for user `05e4d867a47c4c6bbfeeb98041cc1dc1`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:02:11.256677 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-ecf13a41-f935-44c7-895c-25a875cdc6f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:11.260561 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:02:11.267335 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-1b1b337b-f588-4a05-a011-8ae13505387d None None] MFA Rules not processed for user `d567661eb2c74eef9d7ef12176929037`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:02:11.268568 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-1b1b337b-f588-4a05-a011-8ae13505387d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:11.323794 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1309/2584] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:02:11 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:02:11.336197 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecf13a41-f935-44c7-895c-25a875cdc6f5 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:11.336802 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1276/2585] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:02:11 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 09 01:02:11.338021 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-ecf13a41-f935-44c7-895c-25a875cdc6f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:11.341394 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b1b337b-f588-4a05-a011-8ae13505387d None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:11.354447 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:02:11.406571 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b1b337b-f588-4a05-a011-8ae13505387d None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:11.411969 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f5c781c-085f-477d-af67-1b6cb62351f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:11.412603 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f5c781c-085f-477d-af67-1b6cb62351f3 admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:11.412883 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f5c781c-085f-477d-af67-1b6cb62351f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:11.413155 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f5c781c-085f-477d-af67-1b6cb62351f3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:11.413787 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f5c781c-085f-477d-af67-1b6cb62351f3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:11.414043 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecf13a41-f935-44c7-895c-25a875cdc6f5 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:11.415745 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f5c781c-085f-477d-af67-1b6cb62351f3 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:11.419476 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebe03e0a-ea80-4637-b589-9d859729ed22 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a5d95969e74157bb2d2a57207e1651', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:11.420104 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebe03e0a-ea80-4637-b589-9d859729ed22 service nova] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:11.420402 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebe03e0a-ea80-4637-b589-9d859729ed22 service nova] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:11.420653 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebe03e0a-ea80-4637-b589-9d859729ed22 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:11.428497 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebe03e0a-ea80-4637-b589-9d859729ed22 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:11.430181 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebe03e0a-ea80-4637-b589-9d859729ed22 service nova] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:11.440061 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1310/2586] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 01:02:11 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 09 01:02:11.445205 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1277/2587] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 01:02:11 2022] POST /identity/v3/projects => generated 369 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 09 01:02:11.450808 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebe03e0a-ea80-4637-b589-9d859729ed22 service nova] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:11.461796 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebe03e0a-ea80-4637-b589-9d859729ed22 service nova] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:11.519552 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-555c21e2-672f-4010-a9fa-59a05fb4707e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:11.520172 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555c21e2-672f-4010-a9fa-59a05fb4707e admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:11.520447 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555c21e2-672f-4010-a9fa-59a05fb4707e admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:11.520719 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555c21e2-672f-4010-a9fa-59a05fb4707e admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:11.521336 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-555c21e2-672f-4010-a9fa-59a05fb4707e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:11.523542 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-555c21e2-672f-4010-a9fa-59a05fb4707e admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:11.545710 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1311/2588] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 01:02:11 2022] POST /identity/v3/users => generated 291 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 01:02:11.549866 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f5c781c-085f-477d-af67-1b6cb62351f3 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:11.560061 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f5c781c-085f-477d-af67-1b6cb62351f3 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:11.564653 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd2af693-7f89-4a42-b070-55dbe7494c43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:11.565244 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd2af693-7f89-4a42-b070-55dbe7494c43 admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:11.565526 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd2af693-7f89-4a42-b070-55dbe7494c43 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:11.565842 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd2af693-7f89-4a42-b070-55dbe7494c43 admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:11.566477 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd2af693-7f89-4a42-b070-55dbe7494c43 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:11.568100 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd2af693-7f89-4a42-b070-55dbe7494c43 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:11.578257 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1278/2589] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 01:02:11 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 09 01:02:11.583363 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-555c21e2-672f-4010-a9fa-59a05fb4707e admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:11.593893 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-555c21e2-672f-4010-a9fa-59a05fb4707e admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:11.599501 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90954120-70af-4f90-9732-947c39dd40fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:11.600192 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90954120-70af-4f90-9732-947c39dd40fd admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:11.600526 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90954120-70af-4f90-9732-947c39dd40fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:11.600870 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90954120-70af-4f90-9732-947c39dd40fd admin admin] PATH_INFO: `/v3/projects/14ebedb8f9674385bd92ce92b8c3577a/users/fead9f09f85645d8aa62204dcd44d953/roles/092036ee46ac427baf4d56468575f697` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:11.617632 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90954120-70af-4f90-9732-947c39dd40fd admin admin] RBAC: Authorizing `identity:create_grant(project_id=14ebedb8f9674385bd92ce92b8c3577a, user_id=fead9f09f85645d8aa62204dcd44d953, role_id=092036ee46ac427baf4d56468575f697)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:11.620397 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90954120-70af-4f90-9732-947c39dd40fd admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:11.624808 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:02:11.634871 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1312/2590] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 01:02:11 2022] PUT /identity/v3/projects/14ebedb8f9674385bd92ce92b8c3577a/users/fead9f09f85645d8aa62204dcd44d953/roles/092036ee46ac427baf4d56468575f697 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:02:11.639218 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd2af693-7f89-4a42-b070-55dbe7494c43 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:11.649939 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd2af693-7f89-4a42-b070-55dbe7494c43 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:11.719234 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ab3fc2f-0510-4828-8b3b-5602e371d410 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:11.719915 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ab3fc2f-0510-4828-8b3b-5602e371d410 admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:11.720193 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ab3fc2f-0510-4828-8b3b-5602e371d410 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:11.720560 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ab3fc2f-0510-4828-8b3b-5602e371d410 admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:11.721208 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ab3fc2f-0510-4828-8b3b-5602e371d410 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:11.723548 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ab3fc2f-0510-4828-8b3b-5602e371d410 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:11.734530 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1279/2591] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 01:02:11 2022] GET /identity/v3/roles => generated 1420 bytes in 96 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 09 01:02:11.747326 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90954120-70af-4f90-9732-947c39dd40fd admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:11.760181 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90954120-70af-4f90-9732-947c39dd40fd admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:11.765184 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89febb1f-4bcf-4d41-abb1-9113b1bfd625 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:11.765904 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89febb1f-4bcf-4d41-abb1-9113b1bfd625 admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:11.766252 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89febb1f-4bcf-4d41-abb1-9113b1bfd625 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:11.766625 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89febb1f-4bcf-4d41-abb1-9113b1bfd625 admin admin] PATH_INFO: `/v3/projects/14ebedb8f9674385bd92ce92b8c3577a/users/fead9f09f85645d8aa62204dcd44d953/roles/092036ee46ac427baf4d56468575f697` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:11.769472 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89febb1f-4bcf-4d41-abb1-9113b1bfd625 admin admin] RBAC: Authorizing `identity:create_grant(project_id=14ebedb8f9674385bd92ce92b8c3577a, user_id=fead9f09f85645d8aa62204dcd44d953, role_id=092036ee46ac427baf4d56468575f697)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:11.772322 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89febb1f-4bcf-4d41-abb1-9113b1bfd625 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:11.784200 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1313/2592] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 01:02:11 2022] PUT /identity/v3/projects/14ebedb8f9674385bd92ce92b8c3577a/users/fead9f09f85645d8aa62204dcd44d953/roles/092036ee46ac427baf4d56468575f697 => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:02:11.795849 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3636f60f-f1db-40b9-9b75-d26e5a8e4abf None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:11.796137 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3636f60f-f1db-40b9-9b75-d26e5a8e4abf None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:11.796418 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3636f60f-f1db-40b9-9b75-d26e5a8e4abf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:11.829375 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-3636f60f-f1db-40b9-9b75-d26e5a8e4abf None None] MFA Rules not processed for user `fead9f09f85645d8aa62204dcd44d953`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:02:11.830576 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-3636f60f-f1db-40b9-9b75-d26e5a8e4abf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:11.898922 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1280/2593] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:02:11 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:02:11.910318 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89febb1f-4bcf-4d41-abb1-9113b1bfd625 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:11.912068 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-89febb1f-4bcf-4d41-abb1-9113b1bfd625 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:11.969007 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89febb1f-4bcf-4d41-abb1-9113b1bfd625 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:11.972742 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03b80c22-9f92-4bfc-b98f-adbb5dffd81c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3f48335668a4e8f934ac11ff66cd308', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:11.973363 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03b80c22-9f92-4bfc-b98f-adbb5dffd81c service cinder] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:11.973638 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03b80c22-9f92-4bfc-b98f-adbb5dffd81c service cinder] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:11.973905 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03b80c22-9f92-4bfc-b98f-adbb5dffd81c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:11.982982 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03b80c22-9f92-4bfc-b98f-adbb5dffd81c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:11.984529 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03b80c22-9f92-4bfc-b98f-adbb5dffd81c service cinder] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:11.994832 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1314/2594] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 01:02:11 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 01:02:12.040577 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3636f60f-f1db-40b9-9b75-d26e5a8e4abf None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:12.041958 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-3636f60f-f1db-40b9-9b75-d26e5a8e4abf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:12.053788 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3636f60f-f1db-40b9-9b75-d26e5a8e4abf None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:12.057408 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3479180d-7af6-4ac3-a6b8-9681d4e35b19 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3f48335668a4e8f934ac11ff66cd308', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:12.058000 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3479180d-7af6-4ac3-a6b8-9681d4e35b19 service cinder] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:12.058281 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3479180d-7af6-4ac3-a6b8-9681d4e35b19 service cinder] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:12.058558 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3479180d-7af6-4ac3-a6b8-9681d4e35b19 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:12.062575 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:02:12.118465 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3479180d-7af6-4ac3-a6b8-9681d4e35b19 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:12.119929 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3479180d-7af6-4ac3-a6b8-9681d4e35b19 service cinder] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:12.129951 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1281/2595] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 01:02:12 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 01:02:14.119146 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03b80c22-9f92-4bfc-b98f-adbb5dffd81c service cinder] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:14.179030 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03b80c22-9f92-4bfc-b98f-adbb5dffd81c service cinder] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:14.183484 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19bf8496-9600-4256-a4a3-39f4f5a1830d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:14.184128 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19bf8496-9600-4256-a4a3-39f4f5a1830d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:14.184409 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19bf8496-9600-4256-a4a3-39f4f5a1830d admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:14.184682 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19bf8496-9600-4256-a4a3-39f4f5a1830d admin admin] PATH_INFO: `/v3/users/05e4d867a47c4c6bbfeeb98041cc1dc1` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:14.185814 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19bf8496-9600-4256-a4a3-39f4f5a1830d admin admin] RBAC: Authorizing `identity:delete_user(user_id=05e4d867a47c4c6bbfeeb98041cc1dc1)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:14.187760 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19bf8496-9600-4256-a4a3-39f4f5a1830d admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:14.232612 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-19bf8496-9600-4256-a4a3-39f4f5a1830d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '05e4d867a47c4c6bbfeeb98041cc1dc1'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:14.240013 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-19bf8496-9600-4256-a4a3-39f4f5a1830d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '05e4d867a47c4c6bbfeeb98041cc1dc1'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:14.240657 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-19bf8496-9600-4256-a4a3-39f4f5a1830d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '05e4d867a47c4c6bbfeeb98041cc1dc1'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:14.241283 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-19bf8496-9600-4256-a4a3-39f4f5a1830d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '05e4d867a47c4c6bbfeeb98041cc1dc1'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:14.248350 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-19bf8496-9600-4256-a4a3-39f4f5a1830d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '05e4d867a47c4c6bbfeeb98041cc1dc1'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:14.265378 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1315/2596] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 01:02:14 2022] DELETE /identity/v3/users/05e4d867a47c4c6bbfeeb98041cc1dc1 => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:02:14.269808 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb16eedc-1ffb-4dce-bcc8-12a262c8f93b None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:14.270360 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb16eedc-1ffb-4dce-bcc8-12a262c8f93b None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:14.270360 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb16eedc-1ffb-4dce-bcc8-12a262c8f93b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:14.289573 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-bb16eedc-1ffb-4dce-bcc8-12a262c8f93b None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:02:14.290753 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-bb16eedc-1ffb-4dce-bcc8-12a262c8f93b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:14.349741 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1282/2597] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:02:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:02:14.359317 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19bf8496-9600-4256-a4a3-39f4f5a1830d admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:14.360816 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-19bf8496-9600-4256-a4a3-39f4f5a1830d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:14.367310 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:02:14.418933 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19bf8496-9600-4256-a4a3-39f4f5a1830d admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:14.422556 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebe5fd10-f7fe-41d5-8b56-dd06c740f577 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:14.423152 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebe5fd10-f7fe-41d5-8b56-dd06c740f577 service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:14.423432 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebe5fd10-f7fe-41d5-8b56-dd06c740f577 service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:14.423736 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebe5fd10-f7fe-41d5-8b56-dd06c740f577 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:14.432479 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebe5fd10-f7fe-41d5-8b56-dd06c740f577 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:14.433905 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebe5fd10-f7fe-41d5-8b56-dd06c740f577 service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:14.443201 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1316/2598] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 01:02:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 01:02:14.702780 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb16eedc-1ffb-4dce-bcc8-12a262c8f93b None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:14.704213 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-bb16eedc-1ffb-4dce-bcc8-12a262c8f93b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:14.716643 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb16eedc-1ffb-4dce-bcc8-12a262c8f93b None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:14.721842 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58acb19a-60d9-41ff-acfe-bf94aa93ca0a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:14.723332 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58acb19a-60d9-41ff-acfe-bf94aa93ca0a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:14.723651 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58acb19a-60d9-41ff-acfe-bf94aa93ca0a admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:14.723931 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58acb19a-60d9-41ff-acfe-bf94aa93ca0a admin admin] PATH_INFO: `/v3/projects/97b9086fac714440938c6c50bd5026c3` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:14.725101 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58acb19a-60d9-41ff-acfe-bf94aa93ca0a admin admin] RBAC: Authorizing `identity:delete_project(project_id=97b9086fac714440938c6c50bd5026c3)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:14.727063 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58acb19a-60d9-41ff-acfe-bf94aa93ca0a admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:14.773567 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-58acb19a-60d9-41ff-acfe-bf94aa93ca0a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '97b9086fac714440938c6c50bd5026c3', 'request_id': 'req-58acb19a-60d9-41ff-acfe-bf94aa93ca0a'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:14.779255 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-58acb19a-60d9-41ff-acfe-bf94aa93ca0a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '97b9086fac714440938c6c50bd5026c3', 'request_id': 'req-58acb19a-60d9-41ff-acfe-bf94aa93ca0a'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:14.785328 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-58acb19a-60d9-41ff-acfe-bf94aa93ca0a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '97b9086fac714440938c6c50bd5026c3', 'request_id': 'req-58acb19a-60d9-41ff-acfe-bf94aa93ca0a'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:14.797396 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-58acb19a-60d9-41ff-acfe-bf94aa93ca0a admin admin] The token cache is being invalidate because project 97b9086fac714440938c6c50bd5026c3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81453) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 01:02:14.797683 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-58acb19a-60d9-41ff-acfe-bf94aa93ca0a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:14.798291 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-58acb19a-60d9-41ff-acfe-bf94aa93ca0a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:14.799769 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1283/2599] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 01:02:14 2022] DELETE /identity/v3/projects/97b9086fac714440938c6c50bd5026c3 => generated 0 bytes in 98 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:02:14.804676 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a42890-cd2b-417e-9162-d9ebaf0af8d9 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:14.804956 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a42890-cd2b-417e-9162-d9ebaf0af8d9 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:14.805247 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a42890-cd2b-417e-9162-d9ebaf0af8d9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:14.825289 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-45a42890-cd2b-417e-9162-d9ebaf0af8d9 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:02:14.826487 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-45a42890-cd2b-417e-9162-d9ebaf0af8d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:14.887940 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1317/2600] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:02:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:02:14.897695 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9fe5d9d-a007-4c55-a0ef-5b5b56ae3021 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:14.898196 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9fe5d9d-a007-4c55-a0ef-5b5b56ae3021 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:14.898491 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9fe5d9d-a007-4c55-a0ef-5b5b56ae3021 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:14.921179 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-c9fe5d9d-a007-4c55-a0ef-5b5b56ae3021 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:02:14.922518 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-c9fe5d9d-a007-4c55-a0ef-5b5b56ae3021 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:14.930916 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1284/2601] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:02:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:02:14.939677 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45a42890-cd2b-417e-9162-d9ebaf0af8d9 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:14.950205 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45a42890-cd2b-417e-9162-d9ebaf0af8d9 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:14.954987 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec65d6ee-6159-4839-9ccb-164680a5290f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:14.955695 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec65d6ee-6159-4839-9ccb-164680a5290f admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:14.956041 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec65d6ee-6159-4839-9ccb-164680a5290f admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:14.956935 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec65d6ee-6159-4839-9ccb-164680a5290f admin admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:14.957648 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec65d6ee-6159-4839-9ccb-164680a5290f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 01:02:14.958018 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec65d6ee-6159-4839-9ccb-164680a5290f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:14.959733 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec65d6ee-6159-4839-9ccb-164680a5290f admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:14.974716 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1318/2602] 10.222.0.33 () {58 vars in 1153 bytes} [Sun Oct 9 01:02:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 01:02:14.975895 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9fe5d9d-a007-4c55-a0ef-5b5b56ae3021 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:14.985511 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9fe5d9d-a007-4c55-a0ef-5b5b56ae3021 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:14.990406 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed408993-d0ab-429c-8865-7d78a203f95b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:14.991467 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed408993-d0ab-429c-8865-7d78a203f95b admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:14.991773 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed408993-d0ab-429c-8865-7d78a203f95b admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:14.992042 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed408993-d0ab-429c-8865-7d78a203f95b admin admin] PATH_INFO: `/v3/projects` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:14.992667 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed408993-d0ab-429c-8865-7d78a203f95b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:14.994575 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed408993-d0ab-429c-8865-7d78a203f95b admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:15.019899 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:02:15.030344 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1285/2603] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 01:02:14 2022] POST /identity/v3/projects => generated 373 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 01:02:15.031479 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec65d6ee-6159-4839-9ccb-164680a5290f admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:15.041641 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec65d6ee-6159-4839-9ccb-164680a5290f admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:15.104009 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d25ed97-f54e-452a-ab15-3c50aedf2db9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:15.104626 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d25ed97-f54e-452a-ab15-3c50aedf2db9 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:15.104909 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d25ed97-f54e-452a-ab15-3c50aedf2db9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:15.105281 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d25ed97-f54e-452a-ab15-3c50aedf2db9 admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:15.105828 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d25ed97-f54e-452a-ab15-3c50aedf2db9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:15.107798 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d25ed97-f54e-452a-ab15-3c50aedf2db9 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:15.131151 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1319/2604] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 01:02:15 2022] POST /identity/v3/users => generated 287 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 01:02:15.135329 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed408993-d0ab-429c-8865-7d78a203f95b admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:15.145883 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed408993-d0ab-429c-8865-7d78a203f95b admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:15.150460 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d9a6ba7-2c9f-4bfd-9d12-82f7941a61e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:15.151320 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d9a6ba7-2c9f-4bfd-9d12-82f7941a61e6 admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:15.151621 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d9a6ba7-2c9f-4bfd-9d12-82f7941a61e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:15.151901 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d9a6ba7-2c9f-4bfd-9d12-82f7941a61e6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:15.152532 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d9a6ba7-2c9f-4bfd-9d12-82f7941a61e6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:15.154111 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d9a6ba7-2c9f-4bfd-9d12-82f7941a61e6 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:15.165068 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1286/2605] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 01:02:15 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 01:02:15.169388 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d25ed97-f54e-452a-ab15-3c50aedf2db9 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:15.180119 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d25ed97-f54e-452a-ab15-3c50aedf2db9 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:15.185675 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b322186-a334-407d-a756-b2787bd5b4c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:15.186300 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b322186-a334-407d-a756-b2787bd5b4c3 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:15.186577 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b322186-a334-407d-a756-b2787bd5b4c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:15.186854 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b322186-a334-407d-a756-b2787bd5b4c3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:15.187506 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b322186-a334-407d-a756-b2787bd5b4c3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:15.189150 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b322186-a334-407d-a756-b2787bd5b4c3 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:15.201091 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1320/2606] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 01:02:15 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 01:02:15.205519 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d9a6ba7-2c9f-4bfd-9d12-82f7941a61e6 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:15.220080 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d9a6ba7-2c9f-4bfd-9d12-82f7941a61e6 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:15.224838 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e21815b3-8494-4d07-b73a-2a268321b9ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:15.225499 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21815b3-8494-4d07-b73a-2a268321b9ab admin admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:15.225781 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21815b3-8494-4d07-b73a-2a268321b9ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:15.226056 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21815b3-8494-4d07-b73a-2a268321b9ab admin admin] PATH_INFO: `/v3/projects/5562380a08c440759380988a5b2d0b7c/users/082a4dc285c942eca98c4c3ce78193f4/roles/588016933bab436d8b7c3ac76827356d` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:15.242074 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e21815b3-8494-4d07-b73a-2a268321b9ab admin admin] RBAC: Authorizing `identity:create_grant(project_id=5562380a08c440759380988a5b2d0b7c, user_id=082a4dc285c942eca98c4c3ce78193f4, role_id=588016933bab436d8b7c3ac76827356d)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:15.244771 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e21815b3-8494-4d07-b73a-2a268321b9ab admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:15.258439 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1287/2607] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 01:02:15 2022] PUT /identity/v3/projects/5562380a08c440759380988a5b2d0b7c/users/082a4dc285c942eca98c4c3ce78193f4/roles/588016933bab436d8b7c3ac76827356d => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:02:15.268840 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1268c6af-3c23-4166-ba06-eeff9dc54966 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:15.269129 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1268c6af-3c23-4166-ba06-eeff9dc54966 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:15.269403 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1268c6af-3c23-4166-ba06-eeff9dc54966 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:15.295074 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:02:15.299114 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-1268c6af-3c23-4166-ba06-eeff9dc54966 None None] MFA Rules not processed for user `082a4dc285c942eca98c4c3ce78193f4`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:02:15.301836 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-1268c6af-3c23-4166-ba06-eeff9dc54966 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:15.370096 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1321/2608] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:02:15 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:02:15.382789 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e21815b3-8494-4d07-b73a-2a268321b9ab admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:15.384614 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-e21815b3-8494-4d07-b73a-2a268321b9ab admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:15.438754 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1268c6af-3c23-4166-ba06-eeff9dc54966 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:15.440164 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-1268c6af-3c23-4166-ba06-eeff9dc54966 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:15.462399 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e21815b3-8494-4d07-b73a-2a268321b9ab admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:15.468005 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79255c17-9e08-4c5c-95d1-32c050c3efa7 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a5d95969e74157bb2d2a57207e1651', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:15.468882 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79255c17-9e08-4c5c-95d1-32c050c3efa7 service nova] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:15.469204 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79255c17-9e08-4c5c-95d1-32c050c3efa7 service nova] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:15.469485 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79255c17-9e08-4c5c-95d1-32c050c3efa7 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:15.481753 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79255c17-9e08-4c5c-95d1-32c050c3efa7 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:15.483329 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79255c17-9e08-4c5c-95d1-32c050c3efa7 service nova] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:15.493811 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1288/2609] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 01:02:15 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 01:02:15.506211 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1268c6af-3c23-4166-ba06-eeff9dc54966 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:15.514873 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cc5fd39-a308-4bde-ad22-3450e264b3a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:15.514873 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cc5fd39-a308-4bde-ad22-3450e264b3a6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:15.515391 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cc5fd39-a308-4bde-ad22-3450e264b3a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:15.515880 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cc5fd39-a308-4bde-ad22-3450e264b3a6 admin admin] PATH_INFO: `/v3/users/d567661eb2c74eef9d7ef12176929037` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:15.517116 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cc5fd39-a308-4bde-ad22-3450e264b3a6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d567661eb2c74eef9d7ef12176929037)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:15.519104 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cc5fd39-a308-4bde-ad22-3450e264b3a6 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:15.574954 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-7cc5fd39-a308-4bde-ad22-3450e264b3a6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd567661eb2c74eef9d7ef12176929037'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:15.582297 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-7cc5fd39-a308-4bde-ad22-3450e264b3a6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd567661eb2c74eef9d7ef12176929037'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:15.582984 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-7cc5fd39-a308-4bde-ad22-3450e264b3a6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd567661eb2c74eef9d7ef12176929037'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:15.583651 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-7cc5fd39-a308-4bde-ad22-3450e264b3a6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd567661eb2c74eef9d7ef12176929037'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:15.593458 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-7cc5fd39-a308-4bde-ad22-3450e264b3a6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd567661eb2c74eef9d7ef12176929037'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:15.618926 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1322/2610] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 01:02:15 2022] DELETE /identity/v3/users/d567661eb2c74eef9d7ef12176929037 => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 09 01:02:15.623053 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b89681-f45c-425a-9482-c66de0e97914 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:15.623338 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b89681-f45c-425a-9482-c66de0e97914 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:15.623633 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b89681-f45c-425a-9482-c66de0e97914 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:15.642842 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-15b89681-f45c-425a-9482-c66de0e97914 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:02:15.644119 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-15b89681-f45c-425a-9482-c66de0e97914 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:15.708825 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1289/2611] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:02:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:02:15.712680 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cc5fd39-a308-4bde-ad22-3450e264b3a6 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:15.714043 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-7cc5fd39-a308-4bde-ad22-3450e264b3a6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:15.718117 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15b89681-f45c-425a-9482-c66de0e97914 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:15.719443 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-15b89681-f45c-425a-9482-c66de0e97914 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:15.728273 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cc5fd39-a308-4bde-ad22-3450e264b3a6 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:15.732910 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87a767eb-9871-4f6e-aca6-c0e76207b794 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:15.733543 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a767eb-9871-4f6e-aca6-c0e76207b794 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:15.733804 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a767eb-9871-4f6e-aca6-c0e76207b794 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:15.734077 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a767eb-9871-4f6e-aca6-c0e76207b794 admin admin] PATH_INFO: `/v3/users/082a4dc285c942eca98c4c3ce78193f4` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:15.735191 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87a767eb-9871-4f6e-aca6-c0e76207b794 admin admin] RBAC: Authorizing `identity:delete_user(user_id=082a4dc285c942eca98c4c3ce78193f4)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:15.737138 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87a767eb-9871-4f6e-aca6-c0e76207b794 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:15.754316 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:02:15.779795 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15b89681-f45c-425a-9482-c66de0e97914 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:15.783420 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fce3eb9d-e02e-461f-b02b-23dde618a088 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:15.784065 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce3eb9d-e02e-461f-b02b-23dde618a088 service neutron] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:15.784864 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce3eb9d-e02e-461f-b02b-23dde618a088 service neutron] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:15.785155 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce3eb9d-e02e-461f-b02b-23dde618a088 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:15.786630 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-87a767eb-9871-4f6e-aca6-c0e76207b794 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '082a4dc285c942eca98c4c3ce78193f4'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:15.791045 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:02:15.793913 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-87a767eb-9871-4f6e-aca6-c0e76207b794 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '082a4dc285c942eca98c4c3ce78193f4'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:15.794766 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-87a767eb-9871-4f6e-aca6-c0e76207b794 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '082a4dc285c942eca98c4c3ce78193f4'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:15.795426 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-87a767eb-9871-4f6e-aca6-c0e76207b794 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '082a4dc285c942eca98c4c3ce78193f4'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:15.796150 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fce3eb9d-e02e-461f-b02b-23dde618a088 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:15.798013 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fce3eb9d-e02e-461f-b02b-23dde618a088 service neutron] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:15.804044 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-87a767eb-9871-4f6e-aca6-c0e76207b794 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '082a4dc285c942eca98c4c3ce78193f4'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:15.809524 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1290/2612] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 01:02:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 09 01:02:15.824258 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1323/2613] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 01:02:15 2022] DELETE /identity/v3/users/082a4dc285c942eca98c4c3ce78193f4 => generated 0 bytes in 115 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 09 01:02:15.828930 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5fb41da-44a6-467d-a3b5-6acd31542e7d None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:15.829221 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5fb41da-44a6-467d-a3b5-6acd31542e7d None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:15.829553 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5fb41da-44a6-467d-a3b5-6acd31542e7d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:15.849711 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-b5fb41da-44a6-467d-a3b5-6acd31542e7d None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:02:15.850920 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-b5fb41da-44a6-467d-a3b5-6acd31542e7d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:15.915368 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1291/2614] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:02:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 09 01:02:15.925699 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87a767eb-9871-4f6e-aca6-c0e76207b794 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:15.927108 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-87a767eb-9871-4f6e-aca6-c0e76207b794 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:15.991474 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87a767eb-9871-4f6e-aca6-c0e76207b794 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:15.995111 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f268a49e-17cb-452f-8b53-422db085cc17 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:15.995747 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f268a49e-17cb-452f-8b53-422db085cc17 service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:15.996032 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f268a49e-17cb-452f-8b53-422db085cc17 service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:15.996306 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f268a49e-17cb-452f-8b53-422db085cc17 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:16.005996 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f268a49e-17cb-452f-8b53-422db085cc17 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:16.007438 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f268a49e-17cb-452f-8b53-422db085cc17 service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:16.017014 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1324/2615] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 01:02:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 01:02:16.128635 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5fb41da-44a6-467d-a3b5-6acd31542e7d None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:16.130062 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-b5fb41da-44a6-467d-a3b5-6acd31542e7d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:16.149158 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5fb41da-44a6-467d-a3b5-6acd31542e7d None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:16.154213 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf884792-da29-4805-85ae-eefefc803970 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:16.155437 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf884792-da29-4805-85ae-eefefc803970 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:16.155825 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf884792-da29-4805-85ae-eefefc803970 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:16.156095 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf884792-da29-4805-85ae-eefefc803970 admin admin] PATH_INFO: `/v3/projects/d9b18dc3d0f14cb982a654cba1c40532` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:16.157337 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf884792-da29-4805-85ae-eefefc803970 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d9b18dc3d0f14cb982a654cba1c40532)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:16.161094 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf884792-da29-4805-85ae-eefefc803970 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:16.210540 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-cf884792-da29-4805-85ae-eefefc803970 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd9b18dc3d0f14cb982a654cba1c40532', 'request_id': 'req-cf884792-da29-4805-85ae-eefefc803970'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:16.218384 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-cf884792-da29-4805-85ae-eefefc803970 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd9b18dc3d0f14cb982a654cba1c40532', 'request_id': 'req-cf884792-da29-4805-85ae-eefefc803970'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:16.226589 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-cf884792-da29-4805-85ae-eefefc803970 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd9b18dc3d0f14cb982a654cba1c40532', 'request_id': 'req-cf884792-da29-4805-85ae-eefefc803970'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:16.240834 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-cf884792-da29-4805-85ae-eefefc803970 admin admin] The token cache is being invalidate because project d9b18dc3d0f14cb982a654cba1c40532 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81453) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 01:02:16.241125 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-cf884792-da29-4805-85ae-eefefc803970 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:16.241751 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-cf884792-da29-4805-85ae-eefefc803970 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:16.243225 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1292/2616] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 01:02:16 2022] DELETE /identity/v3/projects/d9b18dc3d0f14cb982a654cba1c40532 => generated 0 bytes in 115 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:02:16.247405 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f268a49e-17cb-452f-8b53-422db085cc17 service neutron] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:16.257962 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f268a49e-17cb-452f-8b53-422db085cc17 service neutron] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:16.317196 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38a5d621-4a46-453a-98a1-2be9b0af1c65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:16.317958 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38a5d621-4a46-453a-98a1-2be9b0af1c65 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:16.318340 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38a5d621-4a46-453a-98a1-2be9b0af1c65 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:16.318698 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38a5d621-4a46-453a-98a1-2be9b0af1c65 admin admin] PATH_INFO: `/v3/users/fead9f09f85645d8aa62204dcd44d953` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:16.319911 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38a5d621-4a46-453a-98a1-2be9b0af1c65 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fead9f09f85645d8aa62204dcd44d953)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:16.322111 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38a5d621-4a46-453a-98a1-2be9b0af1c65 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:16.351908 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf884792-da29-4805-85ae-eefefc803970 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:16.353421 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-cf884792-da29-4805-85ae-eefefc803970 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:16.366717 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf884792-da29-4805-85ae-eefefc803970 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:16.371859 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-553a2600-7576-49a1-88e1-8c2bcde666d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:16.372490 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-553a2600-7576-49a1-88e1-8c2bcde666d1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:16.372885 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-553a2600-7576-49a1-88e1-8c2bcde666d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:16.373035 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-553a2600-7576-49a1-88e1-8c2bcde666d1 admin admin] PATH_INFO: `/v3/projects/5562380a08c440759380988a5b2d0b7c` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:16.373494 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:DELETE=14,keystone:SELECT=85,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:02:16.374252 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-553a2600-7576-49a1-88e1-8c2bcde666d1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5562380a08c440759380988a5b2d0b7c)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:16.376097 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-553a2600-7576-49a1-88e1-8c2bcde666d1 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:16.385219 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-38a5d621-4a46-453a-98a1-2be9b0af1c65 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fead9f09f85645d8aa62204dcd44d953'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:16.392798 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-38a5d621-4a46-453a-98a1-2be9b0af1c65 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fead9f09f85645d8aa62204dcd44d953'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:16.393557 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-38a5d621-4a46-453a-98a1-2be9b0af1c65 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fead9f09f85645d8aa62204dcd44d953'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:16.394767 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-38a5d621-4a46-453a-98a1-2be9b0af1c65 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fead9f09f85645d8aa62204dcd44d953'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:16.402430 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-38a5d621-4a46-453a-98a1-2be9b0af1c65 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fead9f09f85645d8aa62204dcd44d953'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:16.421479 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1325/2617] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 01:02:16 2022] DELETE /identity/v3/users/fead9f09f85645d8aa62204dcd44d953 => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:02:16.426004 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-553a2600-7576-49a1-88e1-8c2bcde666d1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5562380a08c440759380988a5b2d0b7c', 'request_id': 'req-553a2600-7576-49a1-88e1-8c2bcde666d1'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:16.431840 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-553a2600-7576-49a1-88e1-8c2bcde666d1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5562380a08c440759380988a5b2d0b7c', 'request_id': 'req-553a2600-7576-49a1-88e1-8c2bcde666d1'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:16.438412 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-553a2600-7576-49a1-88e1-8c2bcde666d1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5562380a08c440759380988a5b2d0b7c', 'request_id': 'req-553a2600-7576-49a1-88e1-8c2bcde666d1'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:16.448259 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:02:16.453152 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-553a2600-7576-49a1-88e1-8c2bcde666d1 admin admin] The token cache is being invalidate because project 5562380a08c440759380988a5b2d0b7c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81453) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 01:02:16.453745 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-553a2600-7576-49a1-88e1-8c2bcde666d1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:16.455003 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-553a2600-7576-49a1-88e1-8c2bcde666d1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:16.457276 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1293/2618] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 01:02:16 2022] DELETE /identity/v3/projects/5562380a08c440759380988a5b2d0b7c => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:02:16.462526 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce58e08-db8b-438e-a947-d0578331487a None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:16.462910 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce58e08-db8b-438e-a947-d0578331487a None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:16.463331 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce58e08-db8b-438e-a947-d0578331487a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:16.485558 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-2ce58e08-db8b-438e-a947-d0578331487a None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:02:16.486883 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-2ce58e08-db8b-438e-a947-d0578331487a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:16.550789 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1326/2619] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:02:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:02:16.558781 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b99688a-8179-40c5-ac8e-02e9ac57db2d None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:16.559073 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b99688a-8179-40c5-ac8e-02e9ac57db2d None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:16.559352 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b99688a-8179-40c5-ac8e-02e9ac57db2d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:16.582690 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-1b99688a-8179-40c5-ac8e-02e9ac57db2d None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:02:16.583993 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-1b99688a-8179-40c5-ac8e-02e9ac57db2d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:16.592606 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1294/2620] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:02:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:02:16.596692 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ce58e08-db8b-438e-a947-d0578331487a None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:16.606339 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ce58e08-db8b-438e-a947-d0578331487a None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:16.610910 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ca2e332-dfbf-4da1-9f9e-dac619142d0b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:16.611507 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ca2e332-dfbf-4da1-9f9e-dac619142d0b admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:16.611835 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ca2e332-dfbf-4da1-9f9e-dac619142d0b admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:16.612109 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ca2e332-dfbf-4da1-9f9e-dac619142d0b admin admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:16.613286 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ca2e332-dfbf-4da1-9f9e-dac619142d0b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 01:02:16.613586 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ca2e332-dfbf-4da1-9f9e-dac619142d0b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:16.615212 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ca2e332-dfbf-4da1-9f9e-dac619142d0b admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:16.627452 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1327/2621] 10.222.0.33 () {58 vars in 1153 bytes} [Sun Oct 9 01:02:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 01:02:16.632431 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b99688a-8179-40c5-ac8e-02e9ac57db2d None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:16.641817 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b99688a-8179-40c5-ac8e-02e9ac57db2d None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:16.646831 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a37b4c6-5302-40a6-bf39-d063a2e1f719 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:16.647454 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a37b4c6-5302-40a6-bf39-d063a2e1f719 admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:16.647761 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a37b4c6-5302-40a6-bf39-d063a2e1f719 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:16.648032 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a37b4c6-5302-40a6-bf39-d063a2e1f719 admin admin] PATH_INFO: `/v3/projects` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:16.648655 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a37b4c6-5302-40a6-bf39-d063a2e1f719 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:16.650538 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a37b4c6-5302-40a6-bf39-d063a2e1f719 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:16.681313 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1295/2622] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 01:02:16 2022] POST /identity/v3/projects => generated 367 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 01:02:16.685680 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ca2e332-dfbf-4da1-9f9e-dac619142d0b admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:16.696386 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ca2e332-dfbf-4da1-9f9e-dac619142d0b admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:16.713200 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a37b4c6-5302-40a6-bf39-d063a2e1f719 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:16.714868 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-4a37b4c6-5302-40a6-bf39-d063a2e1f719 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:16.767084 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91cf51f8-3094-455f-8365-666dd9e810cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:16.767761 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91cf51f8-3094-455f-8365-666dd9e810cb admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:16.768041 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91cf51f8-3094-455f-8365-666dd9e810cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:16.768317 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91cf51f8-3094-455f-8365-666dd9e810cb admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:16.769541 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91cf51f8-3094-455f-8365-666dd9e810cb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:16.771443 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91cf51f8-3094-455f-8365-666dd9e810cb admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:16.788978 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a37b4c6-5302-40a6-bf39-d063a2e1f719 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:16.795620 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1328/2623] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 01:02:16 2022] POST /identity/v3/users => generated 284 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 01:02:16.796982 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-738c9abb-702d-42cd-a8d2-4ced96e23c83 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:16.797354 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91cf51f8-3094-455f-8365-666dd9e810cb admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:16.797622 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738c9abb-702d-42cd-a8d2-4ced96e23c83 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:16.797920 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738c9abb-702d-42cd-a8d2-4ced96e23c83 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:16.798181 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738c9abb-702d-42cd-a8d2-4ced96e23c83 admin admin] PATH_INFO: `/v3/projects/14ebedb8f9674385bd92ce92b8c3577a` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:16.799316 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-738c9abb-702d-42cd-a8d2-4ced96e23c83 admin admin] RBAC: Authorizing `identity:delete_project(project_id=14ebedb8f9674385bd92ce92b8c3577a)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:16.801284 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-738c9abb-702d-42cd-a8d2-4ced96e23c83 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:16.808966 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91cf51f8-3094-455f-8365-666dd9e810cb admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:16.814225 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-732eb04a-123a-47a1-98b0-56cbaf30cfb3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:16.814814 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-732eb04a-123a-47a1-98b0-56cbaf30cfb3 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:16.815091 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-732eb04a-123a-47a1-98b0-56cbaf30cfb3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:16.815363 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-732eb04a-123a-47a1-98b0-56cbaf30cfb3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:16.816061 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-732eb04a-123a-47a1-98b0-56cbaf30cfb3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:16.817818 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-732eb04a-123a-47a1-98b0-56cbaf30cfb3 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:16.828769 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1329/2624] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 01:02:16 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 09 01:02:16.833302 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-732eb04a-123a-47a1-98b0-56cbaf30cfb3 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:16.843180 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-732eb04a-123a-47a1-98b0-56cbaf30cfb3 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:16.850230 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-738c9abb-702d-42cd-a8d2-4ced96e23c83 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '14ebedb8f9674385bd92ce92b8c3577a', 'request_id': 'req-738c9abb-702d-42cd-a8d2-4ced96e23c83'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:16.856046 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-738c9abb-702d-42cd-a8d2-4ced96e23c83 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '14ebedb8f9674385bd92ce92b8c3577a', 'request_id': 'req-738c9abb-702d-42cd-a8d2-4ced96e23c83'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:16.861567 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:DELETE=5,keystone:SELECT=91,keystone:INSERT=4 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:02:16.862364 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-738c9abb-702d-42cd-a8d2-4ced96e23c83 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '14ebedb8f9674385bd92ce92b8c3577a', 'request_id': 'req-738c9abb-702d-42cd-a8d2-4ced96e23c83'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:16.875170 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-738c9abb-702d-42cd-a8d2-4ced96e23c83 admin admin] The token cache is being invalidate because project 14ebedb8f9674385bd92ce92b8c3577a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81453) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 01:02:16.875459 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-738c9abb-702d-42cd-a8d2-4ced96e23c83 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:16.876121 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-738c9abb-702d-42cd-a8d2-4ced96e23c83 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:16.877848 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1296/2625] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 01:02:16 2022] DELETE /identity/v3/projects/14ebedb8f9674385bd92ce92b8c3577a => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:02:16.882411 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36dc1363-c648-4ab1-97a0-f9d110fcd663 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:16.882736 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36dc1363-c648-4ab1-97a0-f9d110fcd663 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:16.883001 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36dc1363-c648-4ab1-97a0-f9d110fcd663 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:16.903559 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-36dc1363-c648-4ab1-97a0-f9d110fcd663 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:02:16.904807 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-36dc1363-c648-4ab1-97a0-f9d110fcd663 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:16.907633 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61b33a6a-3346-408e-b616-fbccca63b835 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:16.908247 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61b33a6a-3346-408e-b616-fbccca63b835 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:16.908527 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61b33a6a-3346-408e-b616-fbccca63b835 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:16.908800 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61b33a6a-3346-408e-b616-fbccca63b835 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:16.910001 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61b33a6a-3346-408e-b616-fbccca63b835 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:16.911571 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61b33a6a-3346-408e-b616-fbccca63b835 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:16.913385 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1297/2626] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:02:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 09 01:02:16.921211 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79a05146-ef95-4f4b-8e40-6197bde2daf6 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:16.921694 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79a05146-ef95-4f4b-8e40-6197bde2daf6 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:16.922040 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79a05146-ef95-4f4b-8e40-6197bde2daf6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:16.922143 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1330/2627] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 01:02:16 2022] GET /identity/v3/roles => generated 1420 bytes in 89 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 09 01:02:16.926199 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61b33a6a-3346-408e-b616-fbccca63b835 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:16.927857 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-61b33a6a-3346-408e-b616-fbccca63b835 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:16.939927 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61b33a6a-3346-408e-b616-fbccca63b835 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:16.943017 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-79a05146-ef95-4f4b-8e40-6197bde2daf6 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:02:16.944164 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-79a05146-ef95-4f4b-8e40-6197bde2daf6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:16.944576 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ec9beb6-3213-44ec-9ced-a181deed7051 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:16.945213 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec9beb6-3213-44ec-9ced-a181deed7051 admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:16.945492 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec9beb6-3213-44ec-9ced-a181deed7051 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:16.945766 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec9beb6-3213-44ec-9ced-a181deed7051 admin admin] PATH_INFO: `/v3/projects/fdc49ce807cc48cdab1aaf875a9b2343/users/db9b5535514649ca9f50cd2e471b0ece/roles/588016933bab436d8b7c3ac76827356d` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:16.951985 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1298/2628] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:02:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 09 01:02:16.956092 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79a05146-ef95-4f4b-8e40-6197bde2daf6 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:16.960950 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ec9beb6-3213-44ec-9ced-a181deed7051 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fdc49ce807cc48cdab1aaf875a9b2343, user_id=db9b5535514649ca9f50cd2e471b0ece, role_id=588016933bab436d8b7c3ac76827356d)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:16.963630 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ec9beb6-3213-44ec-9ced-a181deed7051 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:16.965505 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79a05146-ef95-4f4b-8e40-6197bde2daf6 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:16.970384 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30d763dd-d053-45c3-b91f-fcea205dc4a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:16.971080 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30d763dd-d053-45c3-b91f-fcea205dc4a3 admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:16.971360 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30d763dd-d053-45c3-b91f-fcea205dc4a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:16.971666 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30d763dd-d053-45c3-b91f-fcea205dc4a3 admin admin] PATH_INFO: `/v3/domains` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:16.972967 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30d763dd-d053-45c3-b91f-fcea205dc4a3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 01:02:16.973261 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30d763dd-d053-45c3-b91f-fcea205dc4a3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:16.974828 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30d763dd-d053-45c3-b91f-fcea205dc4a3 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:16.978109 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1331/2629] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 01:02:16 2022] PUT /identity/v3/projects/fdc49ce807cc48cdab1aaf875a9b2343/users/db9b5535514649ca9f50cd2e471b0ece/roles/588016933bab436d8b7c3ac76827356d => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 09 01:02:16.988389 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1299/2630] 10.222.0.33 () {58 vars in 1153 bytes} [Sun Oct 9 01:02:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 01:02:16.989832 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd44bb30-b896-4ed8-920f-e8f0333cfad0 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:16.990174 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd44bb30-b896-4ed8-920f-e8f0333cfad0 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:16.990450 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd44bb30-b896-4ed8-920f-e8f0333cfad0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:16.990834 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30d763dd-d053-45c3-b91f-fcea205dc4a3 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:17.000741 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30d763dd-d053-45c3-b91f-fcea205dc4a3 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:17.018279 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-cd44bb30-b896-4ed8-920f-e8f0333cfad0 None None] MFA Rules not processed for user `db9b5535514649ca9f50cd2e471b0ece`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:02:17.020073 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-cd44bb30-b896-4ed8-920f-e8f0333cfad0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:17.035148 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:02:17.061313 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-843771b8-2f46-4e01-8b72-1bbad8440ff7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:17.061927 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-843771b8-2f46-4e01-8b72-1bbad8440ff7 admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:17.062213 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-843771b8-2f46-4e01-8b72-1bbad8440ff7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:17.062489 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-843771b8-2f46-4e01-8b72-1bbad8440ff7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:17.063129 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-843771b8-2f46-4e01-8b72-1bbad8440ff7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:17.065126 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-843771b8-2f46-4e01-8b72-1bbad8440ff7 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:17.087449 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1332/2631] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:02:16 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:02:17.091927 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd44bb30-b896-4ed8-920f-e8f0333cfad0 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:17.098534 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1300/2632] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 01:02:16 2022] POST /identity/v3/projects => generated 377 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 01:02:17.099655 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-843771b8-2f46-4e01-8b72-1bbad8440ff7 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:17.102549 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd44bb30-b896-4ed8-920f-e8f0333cfad0 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:17.160575 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a052375b-df80-4ae3-8093-0216fedfe634 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:17.161344 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a052375b-df80-4ae3-8093-0216fedfe634 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:17.161667 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-843771b8-2f46-4e01-8b72-1bbad8440ff7 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:17.161905 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a052375b-df80-4ae3-8093-0216fedfe634 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:17.162234 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a052375b-df80-4ae3-8093-0216fedfe634 admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:17.164752 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a052375b-df80-4ae3-8093-0216fedfe634 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:17.166357 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73cbac35-2df3-4a2d-9fe6-d9f857377b5d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:17.166962 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73cbac35-2df3-4a2d-9fe6-d9f857377b5d admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:17.167241 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73cbac35-2df3-4a2d-9fe6-d9f857377b5d admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:17.167514 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73cbac35-2df3-4a2d-9fe6-d9f857377b5d admin admin] PATH_INFO: `/v3/users` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:17.167722 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a052375b-df80-4ae3-8093-0216fedfe634 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:17.168165 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73cbac35-2df3-4a2d-9fe6-d9f857377b5d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:17.170058 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73cbac35-2df3-4a2d-9fe6-d9f857377b5d admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:17.193036 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1301/2633] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 01:02:17 2022] POST /identity/v3/users => generated 289 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 09 01:02:17.199336 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1333/2634] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 01:02:17 2022] POST /identity/v3/projects => generated 365 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 01:02:17.199500 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73cbac35-2df3-4a2d-9fe6-d9f857377b5d admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:17.200638 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a052375b-df80-4ae3-8093-0216fedfe634 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:17.207302 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73cbac35-2df3-4a2d-9fe6-d9f857377b5d admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:17.210987 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:02:17.264319 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68f8892f-d874-43f0-97cd-fe0dd036231d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:17.265035 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68f8892f-d874-43f0-97cd-fe0dd036231d admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:17.265314 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68f8892f-d874-43f0-97cd-fe0dd036231d admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:17.265589 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68f8892f-d874-43f0-97cd-fe0dd036231d admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:17.266118 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a052375b-df80-4ae3-8093-0216fedfe634 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:17.266583 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68f8892f-d874-43f0-97cd-fe0dd036231d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:17.267824 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68f8892f-d874-43f0-97cd-fe0dd036231d admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:17.271429 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00bb18b4-61fb-430d-926f-c35b87161e54 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:17.272059 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00bb18b4-61fb-430d-926f-c35b87161e54 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:17.272353 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00bb18b4-61fb-430d-926f-c35b87161e54 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:17.272613 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00bb18b4-61fb-430d-926f-c35b87161e54 admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:17.273237 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00bb18b4-61fb-430d-926f-c35b87161e54 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:17.275151 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00bb18b4-61fb-430d-926f-c35b87161e54 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:17.278522 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1302/2635] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 01:02:17 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 09 01:02:17.282339 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68f8892f-d874-43f0-97cd-fe0dd036231d admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:17.292679 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68f8892f-d874-43f0-97cd-fe0dd036231d admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:17.299112 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1334/2636] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 01:02:17 2022] POST /identity/v3/users => generated 289 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 09 01:02:17.299256 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49e20051-2947-4d6d-a237-92c0b0eea6f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:17.299256 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e20051-2947-4d6d-a237-92c0b0eea6f6 admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:17.299256 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e20051-2947-4d6d-a237-92c0b0eea6f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:17.299256 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e20051-2947-4d6d-a237-92c0b0eea6f6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:17.299692 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49e20051-2947-4d6d-a237-92c0b0eea6f6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:17.300921 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49e20051-2947-4d6d-a237-92c0b0eea6f6 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:17.301056 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00bb18b4-61fb-430d-926f-c35b87161e54 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:17.310822 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00bb18b4-61fb-430d-926f-c35b87161e54 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:17.311679 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1303/2637] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 01:02:17 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 09 01:02:17.315520 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f962adf8-1f47-4dd2-992b-cad694250b71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:17.316281 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49e20051-2947-4d6d-a237-92c0b0eea6f6 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:17.316649 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f962adf8-1f47-4dd2-992b-cad694250b71 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:17.316932 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f962adf8-1f47-4dd2-992b-cad694250b71 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:17.317209 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f962adf8-1f47-4dd2-992b-cad694250b71 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:17.317843 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f962adf8-1f47-4dd2-992b-cad694250b71 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:17.319437 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f962adf8-1f47-4dd2-992b-cad694250b71 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:17.325882 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49e20051-2947-4d6d-a237-92c0b0eea6f6 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:17.330446 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-defa7a98-3410-4264-a47a-327a6f8cf05d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:17.331073 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-defa7a98-3410-4264-a47a-327a6f8cf05d admin admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:17.331949 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-defa7a98-3410-4264-a47a-327a6f8cf05d admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:17.332226 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-defa7a98-3410-4264-a47a-327a6f8cf05d admin admin] PATH_INFO: `/v3/projects/fac6b6c986e8486fad370ebb71e08461/users/1b7562298fc94c8fbe1ff94cf6752a9c/roles/588016933bab436d8b7c3ac76827356d` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:17.332614 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1335/2638] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 01:02:17 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 09 01:02:17.336691 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f962adf8-1f47-4dd2-992b-cad694250b71 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:17.346344 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f962adf8-1f47-4dd2-992b-cad694250b71 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:17.346618 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-defa7a98-3410-4264-a47a-327a6f8cf05d admin admin] RBAC: Authorizing `identity:create_grant(project_id=fac6b6c986e8486fad370ebb71e08461, user_id=1b7562298fc94c8fbe1ff94cf6752a9c, role_id=588016933bab436d8b7c3ac76827356d)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:17.349454 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-defa7a98-3410-4264-a47a-327a6f8cf05d admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:17.350945 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef0706b3-afd5-4550-b03e-599e3ba9b0d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:17.351654 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef0706b3-afd5-4550-b03e-599e3ba9b0d3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:17.351939 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef0706b3-afd5-4550-b03e-599e3ba9b0d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:17.352214 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef0706b3-afd5-4550-b03e-599e3ba9b0d3 admin admin] PATH_INFO: `/v3/projects/f33d8f9b267a4690b9b8f4caf65222fd/users/5333f15a63fb469abfdbf747e42ca0ae/roles/092036ee46ac427baf4d56468575f697` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:17.362269 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1304/2639] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 01:02:17 2022] PUT /identity/v3/projects/fac6b6c986e8486fad370ebb71e08461/users/1b7562298fc94c8fbe1ff94cf6752a9c/roles/588016933bab436d8b7c3ac76827356d => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:02:17.373965 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-017917ba-a3b2-49d5-af77-88b730a47c26 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:17.374273 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-017917ba-a3b2-49d5-af77-88b730a47c26 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:17.374547 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-017917ba-a3b2-49d5-af77-88b730a47c26 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:17.397198 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:02:17.406576 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-017917ba-a3b2-49d5-af77-88b730a47c26 None None] MFA Rules not processed for user `1b7562298fc94c8fbe1ff94cf6752a9c`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:02:17.408352 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-017917ba-a3b2-49d5-af77-88b730a47c26 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:17.478816 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1305/2640] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:02:17 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 09 01:02:17.483138 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-017917ba-a3b2-49d5-af77-88b730a47c26 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:17.491638 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef0706b3-afd5-4550-b03e-599e3ba9b0d3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f33d8f9b267a4690b9b8f4caf65222fd, user_id=5333f15a63fb469abfdbf747e42ca0ae, role_id=092036ee46ac427baf4d56468575f697)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:17.494330 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef0706b3-afd5-4550-b03e-599e3ba9b0d3 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:17.508484 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1336/2641] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 01:02:17 2022] PUT /identity/v3/projects/f33d8f9b267a4690b9b8f4caf65222fd/users/5333f15a63fb469abfdbf747e42ca0ae/roles/092036ee46ac427baf4d56468575f697 => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 09 01:02:17.512737 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef0706b3-afd5-4550-b03e-599e3ba9b0d3 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:17.549093 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-017917ba-a3b2-49d5-af77-88b730a47c26 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:17.574741 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef0706b3-afd5-4550-b03e-599e3ba9b0d3 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:17.579369 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8458b1c9-0d2b-4f43-8577-4d9a9fa00e3b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:17.580003 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8458b1c9-0d2b-4f43-8577-4d9a9fa00e3b admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:17.580282 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8458b1c9-0d2b-4f43-8577-4d9a9fa00e3b admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:17.580555 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8458b1c9-0d2b-4f43-8577-4d9a9fa00e3b admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:17.581224 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8458b1c9-0d2b-4f43-8577-4d9a9fa00e3b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:17.582858 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8458b1c9-0d2b-4f43-8577-4d9a9fa00e3b admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:17.593507 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1337/2642] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 01:02:17 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 09 01:02:17.597639 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8458b1c9-0d2b-4f43-8577-4d9a9fa00e3b admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:17.607094 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8458b1c9-0d2b-4f43-8577-4d9a9fa00e3b admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:17.607615 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-812d307c-a52b-497f-baa6-3c0198f08295 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:17.608241 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-812d307c-a52b-497f-baa6-3c0198f08295 admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:17.608524 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-812d307c-a52b-497f-baa6-3c0198f08295 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:17.608811 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-812d307c-a52b-497f-baa6-3c0198f08295 admin admin] PATH_INFO: `/v3/projects` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:17.609426 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-812d307c-a52b-497f-baa6-3c0198f08295 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:17.611332 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-812d307c-a52b-497f-baa6-3c0198f08295 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:17.611554 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49c2ea79-8c0c-4b09-b587-2c1cd30be74f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:17.612203 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c2ea79-8c0c-4b09-b587-2c1cd30be74f admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:17.613064 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c2ea79-8c0c-4b09-b587-2c1cd30be74f admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:17.613418 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c2ea79-8c0c-4b09-b587-2c1cd30be74f admin admin] PATH_INFO: `/v3/projects/f33d8f9b267a4690b9b8f4caf65222fd/users/5333f15a63fb469abfdbf747e42ca0ae/roles/092036ee46ac427baf4d56468575f697` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:17.615276 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49c2ea79-8c0c-4b09-b587-2c1cd30be74f admin admin] RBAC: Authorizing `identity:create_grant(project_id=f33d8f9b267a4690b9b8f4caf65222fd, user_id=5333f15a63fb469abfdbf747e42ca0ae, role_id=092036ee46ac427baf4d56468575f697)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:17.617889 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49c2ea79-8c0c-4b09-b587-2c1cd30be74f admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:17.628063 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1338/2643] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 01:02:17 2022] PUT /identity/v3/projects/f33d8f9b267a4690b9b8f4caf65222fd/users/5333f15a63fb469abfdbf747e42ca0ae/roles/092036ee46ac427baf4d56468575f697 => generated 0 bytes in 31 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 09 01:02:17.640033 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ff881f0-c333-49fc-9d9f-dda1a47a16bf None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:17.640310 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ff881f0-c333-49fc-9d9f-dda1a47a16bf None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:17.640574 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ff881f0-c333-49fc-9d9f-dda1a47a16bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:17.643482 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1306/2644] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 01:02:17 2022] POST /identity/v3/projects => generated 379 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 09 01:02:17.644383 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-812d307c-a52b-497f-baa6-3c0198f08295 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:17.669362 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-2ff881f0-c333-49fc-9d9f-dda1a47a16bf None None] MFA Rules not processed for user `5333f15a63fb469abfdbf747e42ca0ae`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:02:17.670322 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-2ff881f0-c333-49fc-9d9f-dda1a47a16bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:17.678312 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:02:17.704691 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:02:17.708479 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-812d307c-a52b-497f-baa6-3c0198f08295 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:17.713551 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d001637-aa3b-4ae5-a5b9-1af1481f7073 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:17.714171 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d001637-aa3b-4ae5-a5b9-1af1481f7073 admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:17.714444 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d001637-aa3b-4ae5-a5b9-1af1481f7073 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:17.714711 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d001637-aa3b-4ae5-a5b9-1af1481f7073 admin admin] PATH_INFO: `/v3/users` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:17.715351 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d001637-aa3b-4ae5-a5b9-1af1481f7073 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:17.717269 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d001637-aa3b-4ae5-a5b9-1af1481f7073 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:17.741593 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1307/2645] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 01:02:17 2022] POST /identity/v3/users => generated 296 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 09 01:02:17.742858 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d001637-aa3b-4ae5-a5b9-1af1481f7073 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:17.743699 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1339/2646] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:02:17 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 09 01:02:17.752839 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d001637-aa3b-4ae5-a5b9-1af1481f7073 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:17.755359 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ff881f0-c333-49fc-9d9f-dda1a47a16bf None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:17.756752 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-2ff881f0-c333-49fc-9d9f-dda1a47a16bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:17.757582 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecf27a8b-ac0b-49ad-84d3-0be5268f1d3b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:17.758174 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf27a8b-ac0b-49ad-84d3-0be5268f1d3b admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:17.758456 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf27a8b-ac0b-49ad-84d3-0be5268f1d3b admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:17.758735 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf27a8b-ac0b-49ad-84d3-0be5268f1d3b admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:17.759386 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecf27a8b-ac0b-49ad-84d3-0be5268f1d3b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:17.760980 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecf27a8b-ac0b-49ad-84d3-0be5268f1d3b admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:17.774657 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1308/2647] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 01:02:17 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 09 01:02:17.775615 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecf27a8b-ac0b-49ad-84d3-0be5268f1d3b admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:17.785829 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecf27a8b-ac0b-49ad-84d3-0be5268f1d3b admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:17.790749 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-744476a4-912f-4bd9-a414-8a8888239534 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:17.791369 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744476a4-912f-4bd9-a414-8a8888239534 admin admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:17.791957 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744476a4-912f-4bd9-a414-8a8888239534 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:17.792240 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744476a4-912f-4bd9-a414-8a8888239534 admin admin] PATH_INFO: `/v3/projects/e0234fb64deb44aa863cd507930e41f3/users/661caa1483a14969972c770053f50c74/roles/092036ee46ac427baf4d56468575f697` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:17.807009 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-744476a4-912f-4bd9-a414-8a8888239534 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e0234fb64deb44aa863cd507930e41f3, user_id=661caa1483a14969972c770053f50c74, role_id=092036ee46ac427baf4d56468575f697)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:17.809683 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-744476a4-912f-4bd9-a414-8a8888239534 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:17.817891 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ff881f0-c333-49fc-9d9f-dda1a47a16bf None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:17.822206 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1309/2648] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 01:02:17 2022] PUT /identity/v3/projects/e0234fb64deb44aa863cd507930e41f3/users/661caa1483a14969972c770053f50c74/roles/092036ee46ac427baf4d56468575f697 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:02:17.826136 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-744476a4-912f-4bd9-a414-8a8888239534 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:17.867930 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea6c75bc-badf-43b2-98b4-ddc79201468e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3f48335668a4e8f934ac11ff66cd308', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:17.868542 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea6c75bc-badf-43b2-98b4-ddc79201468e service cinder] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:17.868836 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea6c75bc-badf-43b2-98b4-ddc79201468e service cinder] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:17.869111 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea6c75bc-badf-43b2-98b4-ddc79201468e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:17.888574 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-744476a4-912f-4bd9-a414-8a8888239534 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:17.894022 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-846756c8-e3f3-45e6-905c-ac27581c3503 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:17.894652 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-846756c8-e3f3-45e6-905c-ac27581c3503 admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:17.894938 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-846756c8-e3f3-45e6-905c-ac27581c3503 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:17.895207 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-846756c8-e3f3-45e6-905c-ac27581c3503 admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:17.895878 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-846756c8-e3f3-45e6-905c-ac27581c3503 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:17.897474 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-846756c8-e3f3-45e6-905c-ac27581c3503 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:17.907738 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1310/2649] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 01:02:17 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 09 01:02:17.912316 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-846756c8-e3f3-45e6-905c-ac27581c3503 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:17.921883 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-846756c8-e3f3-45e6-905c-ac27581c3503 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:17.927288 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50dadd19-4960-4116-96ac-9bd14fe37563 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:17.927995 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50dadd19-4960-4116-96ac-9bd14fe37563 admin admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:17.928283 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50dadd19-4960-4116-96ac-9bd14fe37563 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:17.928562 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50dadd19-4960-4116-96ac-9bd14fe37563 admin admin] PATH_INFO: `/v3/projects/e0234fb64deb44aa863cd507930e41f3/users/661caa1483a14969972c770053f50c74/roles/092036ee46ac427baf4d56468575f697` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:17.929773 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea6c75bc-badf-43b2-98b4-ddc79201468e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:17.930752 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50dadd19-4960-4116-96ac-9bd14fe37563 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e0234fb64deb44aa863cd507930e41f3, user_id=661caa1483a14969972c770053f50c74, role_id=092036ee46ac427baf4d56468575f697)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:17.931213 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea6c75bc-badf-43b2-98b4-ddc79201468e service cinder] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:17.933471 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50dadd19-4960-4116-96ac-9bd14fe37563 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:17.940659 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1340/2650] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 01:02:17 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 186 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 09 01:02:17.946180 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1311/2651] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 01:02:17 2022] PUT /identity/v3/projects/e0234fb64deb44aa863cd507930e41f3/users/661caa1483a14969972c770053f50c74/roles/092036ee46ac427baf4d56468575f697 => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 09 01:02:17.957240 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6778a6ab-9233-4562-9823-45a15742eab8 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:17.957530 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6778a6ab-9233-4562-9823-45a15742eab8 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:17.957804 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6778a6ab-9233-4562-9823-45a15742eab8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:17.986111 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-6778a6ab-9233-4562-9823-45a15742eab8 None None] MFA Rules not processed for user `661caa1483a14969972c770053f50c74`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:02:17.987298 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-6778a6ab-9233-4562-9823-45a15742eab8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:18.017529 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:02:18.025009 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50dadd19-4960-4116-96ac-9bd14fe37563 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:18.058257 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1341/2652] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:02:17 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:02:18.068622 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6778a6ab-9233-4562-9823-45a15742eab8 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:18.069963 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-6778a6ab-9233-4562-9823-45a15742eab8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:18.084334 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50dadd19-4960-4116-96ac-9bd14fe37563 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:18.087985 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-967b3188-fd86-4d1b-bb91-f73f6cba1a3f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3f48335668a4e8f934ac11ff66cd308', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:18.088719 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-967b3188-fd86-4d1b-bb91-f73f6cba1a3f service cinder] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:18.089050 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-967b3188-fd86-4d1b-bb91-f73f6cba1a3f service cinder] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:18.089352 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-967b3188-fd86-4d1b-bb91-f73f6cba1a3f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:18.114380 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:02:18.135131 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6778a6ab-9233-4562-9823-45a15742eab8 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:18.138780 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ce0b87d-0bff-4b66-8894-63d627caa3d5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3f48335668a4e8f934ac11ff66cd308', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:18.139384 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce0b87d-0bff-4b66-8894-63d627caa3d5 service cinder] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:18.139697 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce0b87d-0bff-4b66-8894-63d627caa3d5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:18.139975 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce0b87d-0bff-4b66-8894-63d627caa3d5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:18.147286 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-967b3188-fd86-4d1b-bb91-f73f6cba1a3f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:18.148805 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-967b3188-fd86-4d1b-bb91-f73f6cba1a3f service cinder] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:18.149355 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ce0b87d-0bff-4b66-8894-63d627caa3d5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:18.151074 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ce0b87d-0bff-4b66-8894-63d627caa3d5 service cinder] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:18.158013 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1312/2653] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 01:02:18 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 134 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 09 01:02:18.162492 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1342/2654] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 01:02:18 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 09 01:02:18.547622 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-967b3188-fd86-4d1b-bb91-f73f6cba1a3f service cinder] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:18.609275 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-967b3188-fd86-4d1b-bb91-f73f6cba1a3f service cinder] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:18.613654 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c819c2b-7517-4f34-8625-85cdaa915274 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:18.614278 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c819c2b-7517-4f34-8625-85cdaa915274 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:18.614555 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c819c2b-7517-4f34-8625-85cdaa915274 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:18.614824 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c819c2b-7517-4f34-8625-85cdaa915274 admin admin] PATH_INFO: `/v3/users/1b7562298fc94c8fbe1ff94cf6752a9c` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:18.615967 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c819c2b-7517-4f34-8625-85cdaa915274 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1b7562298fc94c8fbe1ff94cf6752a9c)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:18.617848 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c819c2b-7517-4f34-8625-85cdaa915274 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:18.663163 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-0c819c2b-7517-4f34-8625-85cdaa915274 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1b7562298fc94c8fbe1ff94cf6752a9c'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:18.670039 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-0c819c2b-7517-4f34-8625-85cdaa915274 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1b7562298fc94c8fbe1ff94cf6752a9c'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:18.670688 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-0c819c2b-7517-4f34-8625-85cdaa915274 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1b7562298fc94c8fbe1ff94cf6752a9c'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:18.677145 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-0c819c2b-7517-4f34-8625-85cdaa915274 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1b7562298fc94c8fbe1ff94cf6752a9c'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:18.688968 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-0c819c2b-7517-4f34-8625-85cdaa915274 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1b7562298fc94c8fbe1ff94cf6752a9c'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:18.695221 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1313/2655] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 01:02:18 2022] DELETE /identity/v3/users/1b7562298fc94c8fbe1ff94cf6752a9c => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:02:18.699724 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae1cb20f-637b-40a3-8e56-83b48a19531b None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:18.700181 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae1cb20f-637b-40a3-8e56-83b48a19531b None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:18.700296 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae1cb20f-637b-40a3-8e56-83b48a19531b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:18.720150 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-ae1cb20f-637b-40a3-8e56-83b48a19531b None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:02:18.721342 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-ae1cb20f-637b-40a3-8e56-83b48a19531b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:18.778542 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1343/2656] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:02:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:02:18.788707 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c819c2b-7517-4f34-8625-85cdaa915274 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:18.790145 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-0c819c2b-7517-4f34-8625-85cdaa915274 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:18.838556 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:02:18.850066 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c819c2b-7517-4f34-8625-85cdaa915274 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:18.854159 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7df88a8-02a8-45de-8d6e-b60aa75f05e6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:18.854749 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7df88a8-02a8-45de-8d6e-b60aa75f05e6 service neutron] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:18.855032 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7df88a8-02a8-45de-8d6e-b60aa75f05e6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:18.855307 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7df88a8-02a8-45de-8d6e-b60aa75f05e6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:18.864688 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7df88a8-02a8-45de-8d6e-b60aa75f05e6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:18.866150 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7df88a8-02a8-45de-8d6e-b60aa75f05e6 service neutron] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:18.875658 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1314/2657] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 01:02:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 01:02:19.139969 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae1cb20f-637b-40a3-8e56-83b48a19531b None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:19.141455 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-ae1cb20f-637b-40a3-8e56-83b48a19531b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:19.153649 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae1cb20f-637b-40a3-8e56-83b48a19531b None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:19.158212 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae9ff6c-0854-4d00-9180-85dd2cff2fab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:19.158829 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae9ff6c-0854-4d00-9180-85dd2cff2fab admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:19.159105 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae9ff6c-0854-4d00-9180-85dd2cff2fab admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:19.159527 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae9ff6c-0854-4d00-9180-85dd2cff2fab admin admin] PATH_INFO: `/v3/projects/fac6b6c986e8486fad370ebb71e08461` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:19.160644 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ae9ff6c-0854-4d00-9180-85dd2cff2fab admin admin] RBAC: Authorizing `identity:delete_project(project_id=fac6b6c986e8486fad370ebb71e08461)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:19.162529 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ae9ff6c-0854-4d00-9180-85dd2cff2fab admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:19.213170 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-2ae9ff6c-0854-4d00-9180-85dd2cff2fab admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fac6b6c986e8486fad370ebb71e08461', 'request_id': 'req-2ae9ff6c-0854-4d00-9180-85dd2cff2fab'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:19.218657 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-2ae9ff6c-0854-4d00-9180-85dd2cff2fab admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fac6b6c986e8486fad370ebb71e08461', 'request_id': 'req-2ae9ff6c-0854-4d00-9180-85dd2cff2fab'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:19.220485 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=94,keystone:DELETE=6 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:02:19.226622 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-2ae9ff6c-0854-4d00-9180-85dd2cff2fab admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fac6b6c986e8486fad370ebb71e08461', 'request_id': 'req-2ae9ff6c-0854-4d00-9180-85dd2cff2fab'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:19.239698 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-2ae9ff6c-0854-4d00-9180-85dd2cff2fab admin admin] The token cache is being invalidate because project fac6b6c986e8486fad370ebb71e08461 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 01:02:19.239986 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-2ae9ff6c-0854-4d00-9180-85dd2cff2fab admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:19.240647 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-2ae9ff6c-0854-4d00-9180-85dd2cff2fab admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:19.242140 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1344/2658] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 01:02:19 2022] DELETE /identity/v3/projects/fac6b6c986e8486fad370ebb71e08461 => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:02:19.246269 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7df88a8-02a8-45de-8d6e-b60aa75f05e6 service neutron] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:19.255905 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7df88a8-02a8-45de-8d6e-b60aa75f05e6 service neutron] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:19.312399 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff00d7e7-e229-4003-9c0d-949fcefea272 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:19.313024 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff00d7e7-e229-4003-9c0d-949fcefea272 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:19.313306 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff00d7e7-e229-4003-9c0d-949fcefea272 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:19.313583 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff00d7e7-e229-4003-9c0d-949fcefea272 admin admin] PATH_INFO: `/v3/users/661caa1483a14969972c770053f50c74` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:19.314699 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff00d7e7-e229-4003-9c0d-949fcefea272 admin admin] RBAC: Authorizing `identity:delete_user(user_id=661caa1483a14969972c770053f50c74)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:19.316743 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff00d7e7-e229-4003-9c0d-949fcefea272 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:19.483109 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-ff00d7e7-e229-4003-9c0d-949fcefea272 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '661caa1483a14969972c770053f50c74'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:19.491234 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-ff00d7e7-e229-4003-9c0d-949fcefea272 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '661caa1483a14969972c770053f50c74'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:19.491926 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-ff00d7e7-e229-4003-9c0d-949fcefea272 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '661caa1483a14969972c770053f50c74'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:19.498906 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-ff00d7e7-e229-4003-9c0d-949fcefea272 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '661caa1483a14969972c770053f50c74'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:19.511233 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-ff00d7e7-e229-4003-9c0d-949fcefea272 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '661caa1483a14969972c770053f50c74'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:19.523435 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1315/2659] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 01:02:19 2022] DELETE /identity/v3/users/661caa1483a14969972c770053f50c74 => generated 0 bytes in 277 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:02:19.795135 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae9ff6c-0854-4d00-9180-85dd2cff2fab admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:19.796623 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-2ae9ff6c-0854-4d00-9180-85dd2cff2fab admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:19.812603 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae9ff6c-0854-4d00-9180-85dd2cff2fab admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:19.868806 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdf10e42-4594-4d70-a20b-1da52d8c21e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:19.869421 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf10e42-4594-4d70-a20b-1da52d8c21e8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:19.869691 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf10e42-4594-4d70-a20b-1da52d8c21e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:19.869955 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf10e42-4594-4d70-a20b-1da52d8c21e8 admin admin] PATH_INFO: `/v3/projects/e0234fb64deb44aa863cd507930e41f3` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:19.871031 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdf10e42-4594-4d70-a20b-1da52d8c21e8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e0234fb64deb44aa863cd507930e41f3)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:19.872944 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdf10e42-4594-4d70-a20b-1da52d8c21e8 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:19.918551 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-bdf10e42-4594-4d70-a20b-1da52d8c21e8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e0234fb64deb44aa863cd507930e41f3', 'request_id': 'req-bdf10e42-4594-4d70-a20b-1da52d8c21e8'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:19.924099 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-bdf10e42-4594-4d70-a20b-1da52d8c21e8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e0234fb64deb44aa863cd507930e41f3', 'request_id': 'req-bdf10e42-4594-4d70-a20b-1da52d8c21e8'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:19.930153 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-bdf10e42-4594-4d70-a20b-1da52d8c21e8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e0234fb64deb44aa863cd507930e41f3', 'request_id': 'req-bdf10e42-4594-4d70-a20b-1da52d8c21e8'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:19.943308 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-bdf10e42-4594-4d70-a20b-1da52d8c21e8 admin admin] The token cache is being invalidate because project e0234fb64deb44aa863cd507930e41f3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 01:02:19.943618 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-bdf10e42-4594-4d70-a20b-1da52d8c21e8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:19.944283 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-bdf10e42-4594-4d70-a20b-1da52d8c21e8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:19.945764 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1345/2660] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 01:02:19 2022] DELETE /identity/v3/projects/e0234fb64deb44aa863cd507930e41f3 => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:02:19.950572 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b15fe1-d025-4162-a436-43c312546ec2 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:19.951053 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b15fe1-d025-4162-a436-43c312546ec2 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:19.951192 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b15fe1-d025-4162-a436-43c312546ec2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:19.970724 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-e5b15fe1-d025-4162-a436-43c312546ec2 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:02:19.972536 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-e5b15fe1-d025-4162-a436-43c312546ec2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:20.029322 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1316/2661] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:02:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:02:20.036909 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f275662b-c155-4c2f-b5f5-98be95feba06 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:20.037197 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f275662b-c155-4c2f-b5f5-98be95feba06 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:20.037474 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f275662b-c155-4c2f-b5f5-98be95feba06 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:20.056779 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-f275662b-c155-4c2f-b5f5-98be95feba06 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:02:20.057955 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-f275662b-c155-4c2f-b5f5-98be95feba06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:20.066106 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1346/2662] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:02:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:02:20.070196 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5b15fe1-d025-4162-a436-43c312546ec2 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:20.080405 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5b15fe1-d025-4162-a436-43c312546ec2 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:20.087283 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b40045-6f81-456d-8df3-5f51c6a797a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:20.087918 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b40045-6f81-456d-8df3-5f51c6a797a0 admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:20.088200 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b40045-6f81-456d-8df3-5f51c6a797a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:20.088472 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b40045-6f81-456d-8df3-5f51c6a797a0 admin admin] PATH_INFO: `/v3/domains` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:20.089112 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13b40045-6f81-456d-8df3-5f51c6a797a0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 01:02:20.089412 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13b40045-6f81-456d-8df3-5f51c6a797a0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:20.091049 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13b40045-6f81-456d-8df3-5f51c6a797a0 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:20.092591 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:02:20.105126 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1317/2663] 10.222.0.33 () {58 vars in 1153 bytes} [Sun Oct 9 01:02:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 01:02:20.109438 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f275662b-c155-4c2f-b5f5-98be95feba06 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:20.118553 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f275662b-c155-4c2f-b5f5-98be95feba06 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:20.123130 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f99fecb-a5f6-42eb-9365-7b1d8c86f1e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:20.123745 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f99fecb-a5f6-42eb-9365-7b1d8c86f1e8 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:20.124024 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f99fecb-a5f6-42eb-9365-7b1d8c86f1e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:20.124293 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f99fecb-a5f6-42eb-9365-7b1d8c86f1e8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:20.124920 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f99fecb-a5f6-42eb-9365-7b1d8c86f1e8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:20.126812 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f99fecb-a5f6-42eb-9365-7b1d8c86f1e8 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:20.153708 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1347/2664] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 01:02:20 2022] POST /identity/v3/projects => generated 383 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 01:02:20.157757 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b40045-6f81-456d-8df3-5f51c6a797a0 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:20.167522 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b40045-6f81-456d-8df3-5f51c6a797a0 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:20.222406 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7123bab-5912-4e9f-a204-2e702fb80c8a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:20.223000 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7123bab-5912-4e9f-a204-2e702fb80c8a admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:20.223283 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7123bab-5912-4e9f-a204-2e702fb80c8a admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:20.223557 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7123bab-5912-4e9f-a204-2e702fb80c8a admin admin] PATH_INFO: `/v3/users` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:20.224203 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7123bab-5912-4e9f-a204-2e702fb80c8a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:20.226089 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7123bab-5912-4e9f-a204-2e702fb80c8a admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:20.246632 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1318/2665] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 01:02:20 2022] POST /identity/v3/users => generated 292 bytes in 89 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 09 01:02:20.250732 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f99fecb-a5f6-42eb-9365-7b1d8c86f1e8 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:20.261329 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f99fecb-a5f6-42eb-9365-7b1d8c86f1e8 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:20.265849 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a251116-cff8-48bd-8048-cbd8bdb3e81f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:20.266433 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a251116-cff8-48bd-8048-cbd8bdb3e81f admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:20.266712 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a251116-cff8-48bd-8048-cbd8bdb3e81f admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:20.266983 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a251116-cff8-48bd-8048-cbd8bdb3e81f admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:20.267634 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a251116-cff8-48bd-8048-cbd8bdb3e81f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:20.269869 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a251116-cff8-48bd-8048-cbd8bdb3e81f admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:20.279724 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1348/2666] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 01:02:20 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 01:02:20.283349 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7123bab-5912-4e9f-a204-2e702fb80c8a admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:20.293173 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7123bab-5912-4e9f-a204-2e702fb80c8a admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:20.297893 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1db0c0e9-6de2-4411-a2a6-8cd897c56aca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:20.298472 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db0c0e9-6de2-4411-a2a6-8cd897c56aca admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:20.298752 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db0c0e9-6de2-4411-a2a6-8cd897c56aca admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:20.299029 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db0c0e9-6de2-4411-a2a6-8cd897c56aca admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:20.299691 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1db0c0e9-6de2-4411-a2a6-8cd897c56aca admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:20.301282 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1db0c0e9-6de2-4411-a2a6-8cd897c56aca admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:20.311338 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1319/2667] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 01:02:20 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 01:02:20.315179 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a251116-cff8-48bd-8048-cbd8bdb3e81f admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:20.325778 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a251116-cff8-48bd-8048-cbd8bdb3e81f admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:20.330341 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffc47a73-7b62-4a39-af7c-24a266a37fd1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:20.330963 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc47a73-7b62-4a39-af7c-24a266a37fd1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:20.331848 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc47a73-7b62-4a39-af7c-24a266a37fd1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:20.332124 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc47a73-7b62-4a39-af7c-24a266a37fd1 admin admin] PATH_INFO: `/v3/projects/3b2b4af9a4d64827b537860ce1cd150f/users/215f9e7c85af4ab1914cb4634aaa34c1/roles/588016933bab436d8b7c3ac76827356d` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:20.346233 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffc47a73-7b62-4a39-af7c-24a266a37fd1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3b2b4af9a4d64827b537860ce1cd150f, user_id=215f9e7c85af4ab1914cb4634aaa34c1, role_id=588016933bab436d8b7c3ac76827356d)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:20.348870 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffc47a73-7b62-4a39-af7c-24a266a37fd1 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:20.361265 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1349/2668] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 01:02:20 2022] PUT /identity/v3/projects/3b2b4af9a4d64827b537860ce1cd150f/users/215f9e7c85af4ab1914cb4634aaa34c1/roles/588016933bab436d8b7c3ac76827356d => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:02:20.372817 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7fd1833-4a7d-40e0-be00-c38fb4fc29b4 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:20.373105 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7fd1833-4a7d-40e0-be00-c38fb4fc29b4 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:20.373404 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7fd1833-4a7d-40e0-be00-c38fb4fc29b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:20.401018 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-a7fd1833-4a7d-40e0-be00-c38fb4fc29b4 None None] MFA Rules not processed for user `215f9e7c85af4ab1914cb4634aaa34c1`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:02:20.402216 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-a7fd1833-4a7d-40e0-be00-c38fb4fc29b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:20.463943 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1320/2669] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:02:20 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:02:20.468636 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffc47a73-7b62-4a39-af7c-24a266a37fd1 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:20.485739 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=88,keystone:INSERT=4,keystone:DELETE=8 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:02:20.534391 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffc47a73-7b62-4a39-af7c-24a266a37fd1 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:20.538976 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-291e22c4-f2e7-424c-92ed-368090e7ee2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:20.539575 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-291e22c4-f2e7-424c-92ed-368090e7ee2e admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:20.539883 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-291e22c4-f2e7-424c-92ed-368090e7ee2e admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:20.540153 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-291e22c4-f2e7-424c-92ed-368090e7ee2e admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:20.540784 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-291e22c4-f2e7-424c-92ed-368090e7ee2e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:20.542663 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-291e22c4-f2e7-424c-92ed-368090e7ee2e admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:20.568918 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1350/2670] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 01:02:20 2022] POST /identity/v3/projects => generated 383 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 01:02:20.573065 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7fd1833-4a7d-40e0-be00-c38fb4fc29b4 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:20.583437 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7fd1833-4a7d-40e0-be00-c38fb4fc29b4 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:20.639379 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3f0ea89-4220-4cbe-af81-6ecddc62ed9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:20.639936 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3f0ea89-4220-4cbe-af81-6ecddc62ed9d admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:20.640236 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3f0ea89-4220-4cbe-af81-6ecddc62ed9d admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:20.640489 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3f0ea89-4220-4cbe-af81-6ecddc62ed9d admin admin] PATH_INFO: `/v3/users` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:20.641145 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3f0ea89-4220-4cbe-af81-6ecddc62ed9d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:20.643067 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3f0ea89-4220-4cbe-af81-6ecddc62ed9d admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:20.652059 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:02:20.663950 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1321/2671] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 01:02:20 2022] POST /identity/v3/users => generated 298 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 01:02:20.667914 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-291e22c4-f2e7-424c-92ed-368090e7ee2e admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:20.678557 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-291e22c4-f2e7-424c-92ed-368090e7ee2e admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:20.683127 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd320747-c7c5-4e2c-8f2c-73a8edec140b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:20.683754 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd320747-c7c5-4e2c-8f2c-73a8edec140b admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:20.684037 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd320747-c7c5-4e2c-8f2c-73a8edec140b admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:20.684312 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd320747-c7c5-4e2c-8f2c-73a8edec140b admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:20.684945 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd320747-c7c5-4e2c-8f2c-73a8edec140b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:20.686536 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd320747-c7c5-4e2c-8f2c-73a8edec140b admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:20.697488 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1351/2672] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 01:02:20 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 01:02:20.701389 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3f0ea89-4220-4cbe-af81-6ecddc62ed9d admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:20.711024 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3f0ea89-4220-4cbe-af81-6ecddc62ed9d admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:20.715541 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45b8225e-29e3-466e-bab3-4123f93dd1b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:20.716206 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45b8225e-29e3-466e-bab3-4123f93dd1b1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:20.716488 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45b8225e-29e3-466e-bab3-4123f93dd1b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:20.716763 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45b8225e-29e3-466e-bab3-4123f93dd1b1 admin admin] PATH_INFO: `/v3/projects/c3ad9a857bbc452087cfdabcb9c98aff/users/5f29cf25a9a84ba49bfeb55a2da447d4/roles/092036ee46ac427baf4d56468575f697` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:20.731576 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45b8225e-29e3-466e-bab3-4123f93dd1b1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c3ad9a857bbc452087cfdabcb9c98aff, user_id=5f29cf25a9a84ba49bfeb55a2da447d4, role_id=092036ee46ac427baf4d56468575f697)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:20.734274 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45b8225e-29e3-466e-bab3-4123f93dd1b1 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:20.747009 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1322/2673] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 01:02:20 2022] PUT /identity/v3/projects/c3ad9a857bbc452087cfdabcb9c98aff/users/5f29cf25a9a84ba49bfeb55a2da447d4/roles/092036ee46ac427baf4d56468575f697 => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:02:20.750933 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd320747-c7c5-4e2c-8f2c-73a8edec140b admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:20.760561 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd320747-c7c5-4e2c-8f2c-73a8edec140b admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:20.816543 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fb20e90-73d3-43f8-ad08-06f1d4cd8b8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:20.817149 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fb20e90-73d3-43f8-ad08-06f1d4cd8b8e admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:20.817460 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fb20e90-73d3-43f8-ad08-06f1d4cd8b8e admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:20.817689 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fb20e90-73d3-43f8-ad08-06f1d4cd8b8e admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:20.818546 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fb20e90-73d3-43f8-ad08-06f1d4cd8b8e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:20.820155 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fb20e90-73d3-43f8-ad08-06f1d4cd8b8e admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:20.830201 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1352/2674] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 01:02:20 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 01:02:20.834228 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45b8225e-29e3-466e-bab3-4123f93dd1b1 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:20.844791 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45b8225e-29e3-466e-bab3-4123f93dd1b1 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:20.850170 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe4f7fd7-ca4c-4f36-830a-5e1af392e108 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:20.850812 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe4f7fd7-ca4c-4f36-830a-5e1af392e108 admin admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:20.851085 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe4f7fd7-ca4c-4f36-830a-5e1af392e108 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:20.851356 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe4f7fd7-ca4c-4f36-830a-5e1af392e108 admin admin] PATH_INFO: `/v3/projects/c3ad9a857bbc452087cfdabcb9c98aff/users/5f29cf25a9a84ba49bfeb55a2da447d4/roles/092036ee46ac427baf4d56468575f697` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:20.853277 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe4f7fd7-ca4c-4f36-830a-5e1af392e108 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c3ad9a857bbc452087cfdabcb9c98aff, user_id=5f29cf25a9a84ba49bfeb55a2da447d4, role_id=092036ee46ac427baf4d56468575f697)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:20.855884 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe4f7fd7-ca4c-4f36-830a-5e1af392e108 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:20.866683 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1323/2675] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 01:02:20 2022] PUT /identity/v3/projects/c3ad9a857bbc452087cfdabcb9c98aff/users/5f29cf25a9a84ba49bfeb55a2da447d4/roles/092036ee46ac427baf4d56468575f697 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:02:20.876980 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-585ea006-e9b2-4afe-b993-f75cb3140d03 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:20.877270 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-585ea006-e9b2-4afe-b993-f75cb3140d03 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:20.877557 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-585ea006-e9b2-4afe-b993-f75cb3140d03 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:20.907755 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-585ea006-e9b2-4afe-b993-f75cb3140d03 None None] MFA Rules not processed for user `5f29cf25a9a84ba49bfeb55a2da447d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:02:20.909071 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-585ea006-e9b2-4afe-b993-f75cb3140d03 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:20.987544 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1353/2676] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:02:20 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 09 01:02:20.998596 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe4f7fd7-ca4c-4f36-830a-5e1af392e108 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:21.000477 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-fe4f7fd7-ca4c-4f36-830a-5e1af392e108 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:21.060174 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe4f7fd7-ca4c-4f36-830a-5e1af392e108 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:21.063343 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-290f02ac-a35c-4ed8-8542-a48f1be9e504 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3f48335668a4e8f934ac11ff66cd308', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:21.063961 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-290f02ac-a35c-4ed8-8542-a48f1be9e504 service cinder] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:21.064243 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-290f02ac-a35c-4ed8-8542-a48f1be9e504 service cinder] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:21.064513 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-290f02ac-a35c-4ed8-8542-a48f1be9e504 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:21.074575 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-290f02ac-a35c-4ed8-8542-a48f1be9e504 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:21.076022 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-290f02ac-a35c-4ed8-8542-a48f1be9e504 service cinder] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:21.086273 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1324/2677] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 01:02:20 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 01:02:21.173426 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-585ea006-e9b2-4afe-b993-f75cb3140d03 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:21.183747 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-585ea006-e9b2-4afe-b993-f75cb3140d03 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:21.187284 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47fc898c-4887-4e7b-b285-7059ad988eed service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3f48335668a4e8f934ac11ff66cd308', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:21.187920 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47fc898c-4887-4e7b-b285-7059ad988eed service cinder] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:21.188357 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47fc898c-4887-4e7b-b285-7059ad988eed service cinder] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:21.188633 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47fc898c-4887-4e7b-b285-7059ad988eed service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:21.199325 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:02:21.265328 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47fc898c-4887-4e7b-b285-7059ad988eed service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:21.266755 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47fc898c-4887-4e7b-b285-7059ad988eed service cinder] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:21.275977 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1354/2678] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 01:02:21 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 01:02:21.544072 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-290f02ac-a35c-4ed8-8542-a48f1be9e504 service cinder] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:21.610210 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-290f02ac-a35c-4ed8-8542-a48f1be9e504 service cinder] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:21.614800 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c4f5410-1511-4b79-b879-70c0e3c58056 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:21.615415 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c4f5410-1511-4b79-b879-70c0e3c58056 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:21.615720 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c4f5410-1511-4b79-b879-70c0e3c58056 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:21.615990 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c4f5410-1511-4b79-b879-70c0e3c58056 admin admin] PATH_INFO: `/v3/users/215f9e7c85af4ab1914cb4634aaa34c1` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:21.617084 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c4f5410-1511-4b79-b879-70c0e3c58056 admin admin] RBAC: Authorizing `identity:delete_user(user_id=215f9e7c85af4ab1914cb4634aaa34c1)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:21.618984 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c4f5410-1511-4b79-b879-70c0e3c58056 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:21.674081 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-1c4f5410-1511-4b79-b879-70c0e3c58056 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '215f9e7c85af4ab1914cb4634aaa34c1'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:21.682640 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-1c4f5410-1511-4b79-b879-70c0e3c58056 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '215f9e7c85af4ab1914cb4634aaa34c1'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:21.683309 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-1c4f5410-1511-4b79-b879-70c0e3c58056 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '215f9e7c85af4ab1914cb4634aaa34c1'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:21.686143 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:INSERT=5,keystone:SELECT=88,keystone:DELETE=7 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:02:21.691981 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-1c4f5410-1511-4b79-b879-70c0e3c58056 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '215f9e7c85af4ab1914cb4634aaa34c1'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:21.705887 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-1c4f5410-1511-4b79-b879-70c0e3c58056 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '215f9e7c85af4ab1914cb4634aaa34c1'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:21.712519 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1325/2679] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 01:02:21 2022] DELETE /identity/v3/users/215f9e7c85af4ab1914cb4634aaa34c1 => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:02:21.717313 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a7cf45-f3c6-469d-97d0-66ad5473c90f None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:21.717555 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a7cf45-f3c6-469d-97d0-66ad5473c90f None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:21.717818 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a7cf45-f3c6-469d-97d0-66ad5473c90f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:21.739813 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-55a7cf45-f3c6-469d-97d0-66ad5473c90f None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:02:21.741014 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-55a7cf45-f3c6-469d-97d0-66ad5473c90f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:21.800649 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1355/2680] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:02:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 09 01:02:21.810333 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c4f5410-1511-4b79-b879-70c0e3c58056 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:21.811751 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-1c4f5410-1511-4b79-b879-70c0e3c58056 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:21.869313 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c4f5410-1511-4b79-b879-70c0e3c58056 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:21.872886 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad9b36a2-666d-4a2f-87a8-79ecd9af36b9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:21.873485 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9b36a2-666d-4a2f-87a8-79ecd9af36b9 service neutron] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:21.873763 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9b36a2-666d-4a2f-87a8-79ecd9af36b9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:21.874064 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9b36a2-666d-4a2f-87a8-79ecd9af36b9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:21.883480 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad9b36a2-666d-4a2f-87a8-79ecd9af36b9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:21.884928 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad9b36a2-666d-4a2f-87a8-79ecd9af36b9 service neutron] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:21.895656 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1326/2681] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 01:02:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 01:02:22.166750 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55a7cf45-f3c6-469d-97d0-66ad5473c90f None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:22.168354 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-55a7cf45-f3c6-469d-97d0-66ad5473c90f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:22.181389 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55a7cf45-f3c6-469d-97d0-66ad5473c90f None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:22.185933 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-545cf7f4-e375-46f6-a25b-e42b22d1b817 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:22.186553 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-545cf7f4-e375-46f6-a25b-e42b22d1b817 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:22.186828 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-545cf7f4-e375-46f6-a25b-e42b22d1b817 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:22.187099 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-545cf7f4-e375-46f6-a25b-e42b22d1b817 admin admin] PATH_INFO: `/v3/projects/3b2b4af9a4d64827b537860ce1cd150f` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:22.188667 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-545cf7f4-e375-46f6-a25b-e42b22d1b817 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3b2b4af9a4d64827b537860ce1cd150f)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:22.191136 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-545cf7f4-e375-46f6-a25b-e42b22d1b817 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:22.238070 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-545cf7f4-e375-46f6-a25b-e42b22d1b817 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3b2b4af9a4d64827b537860ce1cd150f', 'request_id': 'req-545cf7f4-e375-46f6-a25b-e42b22d1b817'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:22.243478 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-545cf7f4-e375-46f6-a25b-e42b22d1b817 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3b2b4af9a4d64827b537860ce1cd150f', 'request_id': 'req-545cf7f4-e375-46f6-a25b-e42b22d1b817'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:22.249484 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-545cf7f4-e375-46f6-a25b-e42b22d1b817 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3b2b4af9a4d64827b537860ce1cd150f', 'request_id': 'req-545cf7f4-e375-46f6-a25b-e42b22d1b817'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:22.262228 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-545cf7f4-e375-46f6-a25b-e42b22d1b817 admin admin] The token cache is being invalidate because project 3b2b4af9a4d64827b537860ce1cd150f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 01:02:22.262517 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-545cf7f4-e375-46f6-a25b-e42b22d1b817 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:22.263135 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-545cf7f4-e375-46f6-a25b-e42b22d1b817 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:22.264655 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1356/2682] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 01:02:22 2022] DELETE /identity/v3/projects/3b2b4af9a4d64827b537860ce1cd150f => generated 0 bytes in 98 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:02:22.269181 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad9b36a2-666d-4a2f-87a8-79ecd9af36b9 service neutron] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:22.278840 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad9b36a2-666d-4a2f-87a8-79ecd9af36b9 service neutron] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:22.334242 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-644bc565-9893-446c-a694-a8f830b4f59b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:22.334897 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-644bc565-9893-446c-a694-a8f830b4f59b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:22.335177 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-644bc565-9893-446c-a694-a8f830b4f59b admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:22.335451 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-644bc565-9893-446c-a694-a8f830b4f59b admin admin] PATH_INFO: `/v3/users/5f29cf25a9a84ba49bfeb55a2da447d4` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:22.336595 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-644bc565-9893-446c-a694-a8f830b4f59b admin admin] RBAC: Authorizing `identity:delete_user(user_id=5f29cf25a9a84ba49bfeb55a2da447d4)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:22.338770 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-644bc565-9893-446c-a694-a8f830b4f59b admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:22.382844 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-644bc565-9893-446c-a694-a8f830b4f59b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5f29cf25a9a84ba49bfeb55a2da447d4'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:22.389644 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-644bc565-9893-446c-a694-a8f830b4f59b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5f29cf25a9a84ba49bfeb55a2da447d4'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:22.390285 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-644bc565-9893-446c-a694-a8f830b4f59b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5f29cf25a9a84ba49bfeb55a2da447d4'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:22.398191 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-644bc565-9893-446c-a694-a8f830b4f59b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5f29cf25a9a84ba49bfeb55a2da447d4'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:22.410213 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-644bc565-9893-446c-a694-a8f830b4f59b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5f29cf25a9a84ba49bfeb55a2da447d4'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:22.411647 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:DELETE=13,keystone:SELECT=86,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:02:22.417195 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1327/2683] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 01:02:22 2022] DELETE /identity/v3/users/5f29cf25a9a84ba49bfeb55a2da447d4 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:02:22.684745 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-545cf7f4-e375-46f6-a25b-e42b22d1b817 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:22.686240 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-545cf7f4-e375-46f6-a25b-e42b22d1b817 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:22.702565 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-545cf7f4-e375-46f6-a25b-e42b22d1b817 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:22.734831 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:02:22.762955 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b70cd3b-b625-48af-9f53-3b6e60c85eb0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:22.763574 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b70cd3b-b625-48af-9f53-3b6e60c85eb0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:22.763879 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b70cd3b-b625-48af-9f53-3b6e60c85eb0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:22.764150 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b70cd3b-b625-48af-9f53-3b6e60c85eb0 admin admin] PATH_INFO: `/v3/projects/c3ad9a857bbc452087cfdabcb9c98aff` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:22.765247 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b70cd3b-b625-48af-9f53-3b6e60c85eb0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c3ad9a857bbc452087cfdabcb9c98aff)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:22.767131 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b70cd3b-b625-48af-9f53-3b6e60c85eb0 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:22.815018 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-3b70cd3b-b625-48af-9f53-3b6e60c85eb0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c3ad9a857bbc452087cfdabcb9c98aff', 'request_id': 'req-3b70cd3b-b625-48af-9f53-3b6e60c85eb0'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:22.821514 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-3b70cd3b-b625-48af-9f53-3b6e60c85eb0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c3ad9a857bbc452087cfdabcb9c98aff', 'request_id': 'req-3b70cd3b-b625-48af-9f53-3b6e60c85eb0'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:22.827818 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-3b70cd3b-b625-48af-9f53-3b6e60c85eb0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c3ad9a857bbc452087cfdabcb9c98aff', 'request_id': 'req-3b70cd3b-b625-48af-9f53-3b6e60c85eb0'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:22.840810 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-3b70cd3b-b625-48af-9f53-3b6e60c85eb0 admin admin] The token cache is being invalidate because project c3ad9a857bbc452087cfdabcb9c98aff was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 01:02:22.841104 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-3b70cd3b-b625-48af-9f53-3b6e60c85eb0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:22.841734 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-3b70cd3b-b625-48af-9f53-3b6e60c85eb0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:22.843884 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1357/2684] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 01:02:22 2022] DELETE /identity/v3/projects/c3ad9a857bbc452087cfdabcb9c98aff => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:02:22.849041 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-237ae6fe-a25e-497e-8bc9-22a5447f6552 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:22.849435 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-237ae6fe-a25e-497e-8bc9-22a5447f6552 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:22.849827 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-237ae6fe-a25e-497e-8bc9-22a5447f6552 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:22.876438 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-237ae6fe-a25e-497e-8bc9-22a5447f6552 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:02:22.877655 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-237ae6fe-a25e-497e-8bc9-22a5447f6552 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:22.939452 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1328/2685] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:02:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:02:22.947569 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-237ae6fe-a25e-497e-8bc9-22a5447f6552 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:22.948143 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6955fe2-23c5-4b3a-9b8c-84f4d57cbd7b None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:22.948431 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6955fe2-23c5-4b3a-9b8c-84f4d57cbd7b None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:22.948728 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6955fe2-23c5-4b3a-9b8c-84f4d57cbd7b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:22.948941 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-237ae6fe-a25e-497e-8bc9-22a5447f6552 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:22.960561 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-237ae6fe-a25e-497e-8bc9-22a5447f6552 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:22.965064 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01eeba61-8b78-4eb9-8e39-082d8d6836cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:22.965680 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01eeba61-8b78-4eb9-8e39-082d8d6836cc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:22.965958 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01eeba61-8b78-4eb9-8e39-082d8d6836cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:22.966229 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01eeba61-8b78-4eb9-8e39-082d8d6836cc admin admin] PATH_INFO: `/v3/users/db9b5535514649ca9f50cd2e471b0ece` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:22.967342 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01eeba61-8b78-4eb9-8e39-082d8d6836cc admin admin] RBAC: Authorizing `identity:delete_user(user_id=db9b5535514649ca9f50cd2e471b0ece)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:22.968494 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-b6955fe2-23c5-4b3a-9b8c-84f4d57cbd7b None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:02:22.969281 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01eeba61-8b78-4eb9-8e39-082d8d6836cc admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:22.969688 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-b6955fe2-23c5-4b3a-9b8c-84f4d57cbd7b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:22.978264 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1358/2686] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:02:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 09 01:02:22.982591 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6955fe2-23c5-4b3a-9b8c-84f4d57cbd7b None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:22.992154 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6955fe2-23c5-4b3a-9b8c-84f4d57cbd7b None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:22.997278 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35ddf06f-f40d-43de-9ba8-782e9f482419 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:22.997972 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ddf06f-f40d-43de-9ba8-782e9f482419 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:22.998254 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ddf06f-f40d-43de-9ba8-782e9f482419 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:22.998526 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ddf06f-f40d-43de-9ba8-782e9f482419 admin admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:22.999164 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35ddf06f-f40d-43de-9ba8-782e9f482419 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 01:02:22.999695 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35ddf06f-f40d-43de-9ba8-782e9f482419 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:23.001509 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35ddf06f-f40d-43de-9ba8-782e9f482419 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:23.014076 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1359/2687] 10.222.0.33 () {58 vars in 1153 bytes} [Sun Oct 9 01:02:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 01:02:23.018397 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35ddf06f-f40d-43de-9ba8-782e9f482419 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:23.019915 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-01eeba61-8b78-4eb9-8e39-082d8d6836cc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'db9b5535514649ca9f50cd2e471b0ece'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:23.027475 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-01eeba61-8b78-4eb9-8e39-082d8d6836cc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'db9b5535514649ca9f50cd2e471b0ece'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:23.028195 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-01eeba61-8b78-4eb9-8e39-082d8d6836cc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'db9b5535514649ca9f50cd2e471b0ece'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:23.028333 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35ddf06f-f40d-43de-9ba8-782e9f482419 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:23.033034 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b92749f3-2126-489b-a928-a469cb09f70c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:23.033917 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92749f3-2126-489b-a928-a469cb09f70c admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:23.034872 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92749f3-2126-489b-a928-a469cb09f70c admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:23.035319 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92749f3-2126-489b-a928-a469cb09f70c admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:23.036039 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b92749f3-2126-489b-a928-a469cb09f70c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:23.036832 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-01eeba61-8b78-4eb9-8e39-082d8d6836cc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'db9b5535514649ca9f50cd2e471b0ece'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:23.038157 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b92749f3-2126-489b-a928-a469cb09f70c admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:23.049601 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-01eeba61-8b78-4eb9-8e39-082d8d6836cc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'db9b5535514649ca9f50cd2e471b0ece'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:23.056268 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1329/2688] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 01:02:22 2022] DELETE /identity/v3/users/db9b5535514649ca9f50cd2e471b0ece => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:02:23.061111 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf1f190d-25f6-44f8-8aa6-760e1aae90a2 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:23.061404 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf1f190d-25f6-44f8-8aa6-760e1aae90a2 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:23.061680 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf1f190d-25f6-44f8-8aa6-760e1aae90a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:23.068770 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1360/2689] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 01:02:23 2022] POST /identity/v3/projects => generated 383 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 01:02:23.069732 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b92749f3-2126-489b-a928-a469cb09f70c admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:23.071316 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-b92749f3-2126-489b-a928-a469cb09f70c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:23.084777 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-cf1f190d-25f6-44f8-8aa6-760e1aae90a2 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:02:23.085992 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-cf1f190d-25f6-44f8-8aa6-760e1aae90a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:23.139562 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b92749f3-2126-489b-a928-a469cb09f70c admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:23.144525 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1923fdc-3a5f-4d75-aa56-c3d1bc1e572d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:23.145120 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1923fdc-3a5f-4d75-aa56-c3d1bc1e572d admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:23.145424 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1923fdc-3a5f-4d75-aa56-c3d1bc1e572d admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:23.145698 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1923fdc-3a5f-4d75-aa56-c3d1bc1e572d admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:23.149208 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1330/2690] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:02:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 09 01:02:23.149366 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1923fdc-3a5f-4d75-aa56-c3d1bc1e572d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:23.149366 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1923fdc-3a5f-4d75-aa56-c3d1bc1e572d admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:23.156462 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf1f190d-25f6-44f8-8aa6-760e1aae90a2 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:23.157913 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-cf1f190d-25f6-44f8-8aa6-760e1aae90a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:23.172871 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1361/2691] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 01:02:23 2022] POST /identity/v3/users => generated 292 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 09 01:02:23.174298 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1923fdc-3a5f-4d75-aa56-c3d1bc1e572d admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:23.176153 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:02:23.185507 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1923fdc-3a5f-4d75-aa56-c3d1bc1e572d admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:23.190035 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57ad0a57-7414-437d-9f81-3067a4a003ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:23.190737 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57ad0a57-7414-437d-9f81-3067a4a003ed admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:23.191021 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57ad0a57-7414-437d-9f81-3067a4a003ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:23.191297 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57ad0a57-7414-437d-9f81-3067a4a003ed admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:23.191967 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57ad0a57-7414-437d-9f81-3067a4a003ed admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:23.193569 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57ad0a57-7414-437d-9f81-3067a4a003ed admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:23.200905 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=88,keystone:DELETE=7,keystone:INSERT=5 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:02:23.207767 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1362/2692] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 01:02:23 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 09 01:02:23.211802 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57ad0a57-7414-437d-9f81-3067a4a003ed admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:23.223229 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57ad0a57-7414-437d-9f81-3067a4a003ed admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:23.223921 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf1f190d-25f6-44f8-8aa6-760e1aae90a2 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:23.227512 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd8f6718-6a79-4796-9b2b-189c43626740 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:23.228145 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a6db1a4-3751-417c-8c31-1efc6be2f6ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:23.228325 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd8f6718-6a79-4796-9b2b-189c43626740 service neutron] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:23.228460 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd8f6718-6a79-4796-9b2b-189c43626740 service neutron] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:23.228731 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd8f6718-6a79-4796-9b2b-189c43626740 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:23.228860 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a6db1a4-3751-417c-8c31-1efc6be2f6ed admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:23.229033 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a6db1a4-3751-417c-8c31-1efc6be2f6ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:23.229310 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a6db1a4-3751-417c-8c31-1efc6be2f6ed admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:23.229940 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a6db1a4-3751-417c-8c31-1efc6be2f6ed admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:23.231585 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a6db1a4-3751-417c-8c31-1efc6be2f6ed admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:23.238582 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd8f6718-6a79-4796-9b2b-189c43626740 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:23.240061 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd8f6718-6a79-4796-9b2b-189c43626740 service neutron] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:23.242864 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1363/2693] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 01:02:23 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 01:02:23.246882 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a6db1a4-3751-417c-8c31-1efc6be2f6ed admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:23.250311 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1331/2694] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 01:02:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 09 01:02:23.256260 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a6db1a4-3751-417c-8c31-1efc6be2f6ed admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:23.260836 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2a9e719-30f8-4e83-b92a-c9b1321037a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:23.261472 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2a9e719-30f8-4e83-b92a-c9b1321037a7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:23.261749 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2a9e719-30f8-4e83-b92a-c9b1321037a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:23.262023 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2a9e719-30f8-4e83-b92a-c9b1321037a7 admin admin] PATH_INFO: `/v3/projects/6b037d8bc6e240f7a981facfd03c639d/users/5c28bc0f5b33408ba69a74712524f8d9/roles/588016933bab436d8b7c3ac76827356d` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:23.276717 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2a9e719-30f8-4e83-b92a-c9b1321037a7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6b037d8bc6e240f7a981facfd03c639d, user_id=5c28bc0f5b33408ba69a74712524f8d9, role_id=588016933bab436d8b7c3ac76827356d)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:23.279350 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2a9e719-30f8-4e83-b92a-c9b1321037a7 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:23.293209 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1364/2695] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 01:02:23 2022] PUT /identity/v3/projects/6b037d8bc6e240f7a981facfd03c639d/users/5c28bc0f5b33408ba69a74712524f8d9/roles/588016933bab436d8b7c3ac76827356d => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 09 01:02:23.361615 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08dbe1a3-6208-4856-98a0-6bc24beb8ac3 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:23.362167 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08dbe1a3-6208-4856-98a0-6bc24beb8ac3 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:23.362167 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08dbe1a3-6208-4856-98a0-6bc24beb8ac3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:23.389778 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-08dbe1a3-6208-4856-98a0-6bc24beb8ac3 None None] MFA Rules not processed for user `5c28bc0f5b33408ba69a74712524f8d9`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:02:23.390989 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-08dbe1a3-6208-4856-98a0-6bc24beb8ac3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:23.459362 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1332/2696] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:02:23 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:02:23.463979 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2a9e719-30f8-4e83-b92a-c9b1321037a7 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:23.534870 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2a9e719-30f8-4e83-b92a-c9b1321037a7 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:23.537665 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08dbe1a3-6208-4856-98a0-6bc24beb8ac3 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:23.539966 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b4a2f2-80f1-4ff1-bd1c-e10ad3637737 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:23.540339 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b4a2f2-80f1-4ff1-bd1c-e10ad3637737 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:23.540617 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b4a2f2-80f1-4ff1-bd1c-e10ad3637737 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:23.540888 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b4a2f2-80f1-4ff1-bd1c-e10ad3637737 admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:23.541239 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-08dbe1a3-6208-4856-98a0-6bc24beb8ac3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:23.541510 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41b4a2f2-80f1-4ff1-bd1c-e10ad3637737 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:23.543399 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41b4a2f2-80f1-4ff1-bd1c-e10ad3637737 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:23.558725 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08dbe1a3-6208-4856-98a0-6bc24beb8ac3 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:23.564631 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74086034-5a2a-46ab-b48e-23cf06325f91 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:23.565666 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74086034-5a2a-46ab-b48e-23cf06325f91 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:23.566069 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74086034-5a2a-46ab-b48e-23cf06325f91 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:23.566447 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74086034-5a2a-46ab-b48e-23cf06325f91 admin admin] PATH_INFO: `/v3/projects/fdc49ce807cc48cdab1aaf875a9b2343` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:23.568204 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74086034-5a2a-46ab-b48e-23cf06325f91 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fdc49ce807cc48cdab1aaf875a9b2343)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:23.571014 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74086034-5a2a-46ab-b48e-23cf06325f91 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:23.571602 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1365/2697] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 01:02:23 2022] POST /identity/v3/projects => generated 385 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 01:02:23.576105 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b4a2f2-80f1-4ff1-bd1c-e10ad3637737 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:23.623226 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-74086034-5a2a-46ab-b48e-23cf06325f91 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fdc49ce807cc48cdab1aaf875a9b2343', 'request_id': 'req-74086034-5a2a-46ab-b48e-23cf06325f91'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:23.628879 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-74086034-5a2a-46ab-b48e-23cf06325f91 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fdc49ce807cc48cdab1aaf875a9b2343', 'request_id': 'req-74086034-5a2a-46ab-b48e-23cf06325f91'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:23.635161 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-74086034-5a2a-46ab-b48e-23cf06325f91 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fdc49ce807cc48cdab1aaf875a9b2343', 'request_id': 'req-74086034-5a2a-46ab-b48e-23cf06325f91'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:23.641941 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b4a2f2-80f1-4ff1-bd1c-e10ad3637737 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:23.648993 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-74086034-5a2a-46ab-b48e-23cf06325f91 admin admin] The token cache is being invalidate because project fdc49ce807cc48cdab1aaf875a9b2343 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81453) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 01:02:23.649283 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-74086034-5a2a-46ab-b48e-23cf06325f91 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:23.649918 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-74086034-5a2a-46ab-b48e-23cf06325f91 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:23.651713 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1333/2698] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 01:02:23 2022] DELETE /identity/v3/projects/fdc49ce807cc48cdab1aaf875a9b2343 => generated 0 bytes in 115 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 09 01:02:23.655460 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74086034-5a2a-46ab-b48e-23cf06325f91 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:23.656792 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-74086034-5a2a-46ab-b48e-23cf06325f91 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:23.687352 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:02:23.693759 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:02:23.700624 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4024f998-f59a-44a4-94bd-c60b72c7fb96 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:23.701587 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4024f998-f59a-44a4-94bd-c60b72c7fb96 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:23.702433 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4024f998-f59a-44a4-94bd-c60b72c7fb96 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:23.702783 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4024f998-f59a-44a4-94bd-c60b72c7fb96 admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:23.703411 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4024f998-f59a-44a4-94bd-c60b72c7fb96 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:23.705384 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4024f998-f59a-44a4-94bd-c60b72c7fb96 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:23.724337 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74086034-5a2a-46ab-b48e-23cf06325f91 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:23.727347 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1366/2699] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 01:02:23 2022] POST /identity/v3/users => generated 299 bytes in 152 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 09 01:02:23.729183 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7ee9012-98b2-4e56-afa1-d759c362c1ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:23.729883 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ee9012-98b2-4e56-afa1-d759c362c1ed admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:23.730230 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ee9012-98b2-4e56-afa1-d759c362c1ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:23.730587 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ee9012-98b2-4e56-afa1-d759c362c1ed admin admin] PATH_INFO: `/v3/users/5333f15a63fb469abfdbf747e42ca0ae` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:23.731396 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4024f998-f59a-44a4-94bd-c60b72c7fb96 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:23.731909 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7ee9012-98b2-4e56-afa1-d759c362c1ed admin admin] RBAC: Authorizing `identity:delete_user(user_id=5333f15a63fb469abfdbf747e42ca0ae)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:23.732992 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-4024f998-f59a-44a4-94bd-c60b72c7fb96 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:23.733979 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7ee9012-98b2-4e56-afa1-d759c362c1ed admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:23.744824 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4024f998-f59a-44a4-94bd-c60b72c7fb96 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:23.749417 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f78a32ff-6072-46d9-9363-7fd2f75f6204 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:23.749983 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f78a32ff-6072-46d9-9363-7fd2f75f6204 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:23.750255 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f78a32ff-6072-46d9-9363-7fd2f75f6204 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:23.750524 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f78a32ff-6072-46d9-9363-7fd2f75f6204 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:23.751202 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f78a32ff-6072-46d9-9363-7fd2f75f6204 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:23.753609 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f78a32ff-6072-46d9-9363-7fd2f75f6204 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:23.764051 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1367/2700] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 01:02:23 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 09 01:02:23.769372 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f78a32ff-6072-46d9-9363-7fd2f75f6204 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:23.779069 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f78a32ff-6072-46d9-9363-7fd2f75f6204 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:23.781849 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-f7ee9012-98b2-4e56-afa1-d759c362c1ed admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5333f15a63fb469abfdbf747e42ca0ae'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:23.783915 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eed5811c-cb42-4c23-8ec3-bb6a06bac36d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:23.784678 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed5811c-cb42-4c23-8ec3-bb6a06bac36d admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:23.784961 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed5811c-cb42-4c23-8ec3-bb6a06bac36d admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:23.785236 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed5811c-cb42-4c23-8ec3-bb6a06bac36d admin admin] PATH_INFO: `/v3/projects/ff44ced9f1f241ad8b78680960b60f01/users/4cd6a2e4d308476d9c69fd5a95cb2ae3/roles/092036ee46ac427baf4d56468575f697` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:23.789273 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-f7ee9012-98b2-4e56-afa1-d759c362c1ed admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5333f15a63fb469abfdbf747e42ca0ae'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:23.789943 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-f7ee9012-98b2-4e56-afa1-d759c362c1ed admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5333f15a63fb469abfdbf747e42ca0ae'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:23.797939 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-f7ee9012-98b2-4e56-afa1-d759c362c1ed admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5333f15a63fb469abfdbf747e42ca0ae'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:23.800291 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eed5811c-cb42-4c23-8ec3-bb6a06bac36d admin admin] RBAC: Authorizing `identity:create_grant(project_id=ff44ced9f1f241ad8b78680960b60f01, user_id=4cd6a2e4d308476d9c69fd5a95cb2ae3, role_id=092036ee46ac427baf4d56468575f697)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:23.803021 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eed5811c-cb42-4c23-8ec3-bb6a06bac36d admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:23.810539 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-f7ee9012-98b2-4e56-afa1-d759c362c1ed admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5333f15a63fb469abfdbf747e42ca0ae'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:23.816558 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1368/2701] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 01:02:23 2022] PUT /identity/v3/projects/ff44ced9f1f241ad8b78680960b60f01/users/4cd6a2e4d308476d9c69fd5a95cb2ae3/roles/092036ee46ac427baf4d56468575f697 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 09 01:02:23.820193 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1334/2702] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 01:02:23 2022] DELETE /identity/v3/users/5333f15a63fb469abfdbf747e42ca0ae => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 09 01:02:23.823519 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eed5811c-cb42-4c23-8ec3-bb6a06bac36d admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:23.825187 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-eed5811c-cb42-4c23-8ec3-bb6a06bac36d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:23.891003 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eed5811c-cb42-4c23-8ec3-bb6a06bac36d admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:23.948752 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57fceaf1-d15d-4a83-8388-363104b8dcef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:23.949359 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57fceaf1-d15d-4a83-8388-363104b8dcef admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:23.949641 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57fceaf1-d15d-4a83-8388-363104b8dcef admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:23.949915 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57fceaf1-d15d-4a83-8388-363104b8dcef admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:23.950554 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57fceaf1-d15d-4a83-8388-363104b8dcef admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:23.952185 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57fceaf1-d15d-4a83-8388-363104b8dcef admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:23.972490 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1369/2703] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 01:02:23 2022] GET /identity/v3/roles => generated 1420 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 09 01:02:23.976745 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7ee9012-98b2-4e56-afa1-d759c362c1ed admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:23.986744 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7ee9012-98b2-4e56-afa1-d759c362c1ed admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:23.991461 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09e1eb9d-2916-4eb7-9954-f154f15775c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:23.992153 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09e1eb9d-2916-4eb7-9954-f154f15775c6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:23.992433 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09e1eb9d-2916-4eb7-9954-f154f15775c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:23.992710 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09e1eb9d-2916-4eb7-9954-f154f15775c6 admin admin] PATH_INFO: `/v3/projects/ff44ced9f1f241ad8b78680960b60f01/users/4cd6a2e4d308476d9c69fd5a95cb2ae3/roles/092036ee46ac427baf4d56468575f697` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:23.994608 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09e1eb9d-2916-4eb7-9954-f154f15775c6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ff44ced9f1f241ad8b78680960b60f01, user_id=4cd6a2e4d308476d9c69fd5a95cb2ae3, role_id=092036ee46ac427baf4d56468575f697)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:23.997289 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09e1eb9d-2916-4eb7-9954-f154f15775c6 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:24.008490 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1335/2704] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 01:02:23 2022] PUT /identity/v3/projects/ff44ced9f1f241ad8b78680960b60f01/users/4cd6a2e4d308476d9c69fd5a95cb2ae3/roles/092036ee46ac427baf4d56468575f697 => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 09 01:02:24.019426 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d3e906-ec83-49ca-b69d-ad3efbba8f25 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:24.019791 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d3e906-ec83-49ca-b69d-ad3efbba8f25 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:24.020071 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d3e906-ec83-49ca-b69d-ad3efbba8f25 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:24.048997 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-12d3e906-ec83-49ca-b69d-ad3efbba8f25 None None] MFA Rules not processed for user `4cd6a2e4d308476d9c69fd5a95cb2ae3`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:02:24.050394 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-12d3e906-ec83-49ca-b69d-ad3efbba8f25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:24.091822 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09e1eb9d-2916-4eb7-9954-f154f15775c6 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:24.093666 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-09e1eb9d-2916-4eb7-9954-f154f15775c6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:24.103709 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:02:24.134650 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1370/2705] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:02:24 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:02:24.146829 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12d3e906-ec83-49ca-b69d-ad3efbba8f25 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:24.148318 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-12d3e906-ec83-49ca-b69d-ad3efbba8f25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:24.169130 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09e1eb9d-2916-4eb7-9954-f154f15775c6 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:24.174592 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eacf7ce-6c34-4bb2-bace-eec891a7ac2c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:24.175789 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eacf7ce-6c34-4bb2-bace-eec891a7ac2c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:24.176082 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eacf7ce-6c34-4bb2-bace-eec891a7ac2c admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:24.176749 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eacf7ce-6c34-4bb2-bace-eec891a7ac2c admin admin] PATH_INFO: `/v3/projects/f33d8f9b267a4690b9b8f4caf65222fd` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:24.177886 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eacf7ce-6c34-4bb2-bace-eec891a7ac2c admin admin] RBAC: Authorizing `identity:delete_project(project_id=f33d8f9b267a4690b9b8f4caf65222fd)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:24.179807 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eacf7ce-6c34-4bb2-bace-eec891a7ac2c admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:24.205549 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12d3e906-ec83-49ca-b69d-ad3efbba8f25 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:24.209373 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fa8c415-9244-4e66-b214-9f79109adc98 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3f48335668a4e8f934ac11ff66cd308', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:24.209970 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa8c415-9244-4e66-b214-9f79109adc98 service cinder] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:24.210251 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa8c415-9244-4e66-b214-9f79109adc98 service cinder] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:24.210578 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa8c415-9244-4e66-b214-9f79109adc98 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:24.212583 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-6fa8c415-9244-4e66-b214-9f79109adc98 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:24.217584 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:02:24.228843 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-2eacf7ce-6c34-4bb2-bace-eec891a7ac2c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f33d8f9b267a4690b9b8f4caf65222fd', 'request_id': 'req-2eacf7ce-6c34-4bb2-bace-eec891a7ac2c'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:24.234295 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-2eacf7ce-6c34-4bb2-bace-eec891a7ac2c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f33d8f9b267a4690b9b8f4caf65222fd', 'request_id': 'req-2eacf7ce-6c34-4bb2-bace-eec891a7ac2c'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:24.241317 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-2eacf7ce-6c34-4bb2-bace-eec891a7ac2c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f33d8f9b267a4690b9b8f4caf65222fd', 'request_id': 'req-2eacf7ce-6c34-4bb2-bace-eec891a7ac2c'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:24.253851 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-2eacf7ce-6c34-4bb2-bace-eec891a7ac2c admin admin] The token cache is being invalidate because project f33d8f9b267a4690b9b8f4caf65222fd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81453) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 01:02:24.254135 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-2eacf7ce-6c34-4bb2-bace-eec891a7ac2c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:24.254766 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-2eacf7ce-6c34-4bb2-bace-eec891a7ac2c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:24.256515 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1336/2706] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 01:02:24 2022] DELETE /identity/v3/projects/f33d8f9b267a4690b9b8f4caf65222fd => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:02:24.262565 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90557325-48d5-4f65-8bad-b35188809f80 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:24.262851 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90557325-48d5-4f65-8bad-b35188809f80 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:24.263123 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90557325-48d5-4f65-8bad-b35188809f80 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:24.275447 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fa8c415-9244-4e66-b214-9f79109adc98 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:24.277201 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fa8c415-9244-4e66-b214-9f79109adc98 service cinder] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:24.283170 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-90557325-48d5-4f65-8bad-b35188809f80 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:02:24.284402 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-90557325-48d5-4f65-8bad-b35188809f80 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:24.286494 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1371/2707] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 01:02:24 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 140 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 09 01:02:24.344932 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1337/2708] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:02:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 09 01:02:24.354026 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1df7171-a1be-4ce8-af12-38b0f09ea36b None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:24.354318 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1df7171-a1be-4ce8-af12-38b0f09ea36b None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:24.354600 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1df7171-a1be-4ce8-af12-38b0f09ea36b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:24.375654 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-d1df7171-a1be-4ce8-af12-38b0f09ea36b None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:02:24.376845 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-d1df7171-a1be-4ce8-af12-38b0f09ea36b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:24.385303 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1372/2709] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:02:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:02:24.389503 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90557325-48d5-4f65-8bad-b35188809f80 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:24.399084 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90557325-48d5-4f65-8bad-b35188809f80 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:24.404939 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abd96890-64aa-4dde-ac23-17086cfb65c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:24.405537 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd96890-64aa-4dde-ac23-17086cfb65c4 admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:24.405817 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd96890-64aa-4dde-ac23-17086cfb65c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:24.406090 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd96890-64aa-4dde-ac23-17086cfb65c4 admin admin] PATH_INFO: `/v3/domains` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:24.406728 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abd96890-64aa-4dde-ac23-17086cfb65c4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 01:02:24.407036 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abd96890-64aa-4dde-ac23-17086cfb65c4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:24.408639 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abd96890-64aa-4dde-ac23-17086cfb65c4 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:24.422229 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1338/2710] 10.222.0.33 () {58 vars in 1153 bytes} [Sun Oct 9 01:02:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 01:02:24.424369 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1df7171-a1be-4ce8-af12-38b0f09ea36b None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:24.434042 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1df7171-a1be-4ce8-af12-38b0f09ea36b None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:24.438668 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaa5397a-4022-4557-900b-4d76bd825d7f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:24.439267 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaa5397a-4022-4557-900b-4d76bd825d7f admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:24.439547 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaa5397a-4022-4557-900b-4d76bd825d7f admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:24.439863 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaa5397a-4022-4557-900b-4d76bd825d7f admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:24.441042 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaa5397a-4022-4557-900b-4d76bd825d7f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:24.442927 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaa5397a-4022-4557-900b-4d76bd825d7f admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:24.469550 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1373/2711] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 01:02:24 2022] POST /identity/v3/projects => generated 375 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 01:02:24.475059 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abd96890-64aa-4dde-ac23-17086cfb65c4 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:24.486810 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abd96890-64aa-4dde-ac23-17086cfb65c4 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:24.546777 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27fe6428-6096-49b8-9e99-cd5679f18552 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:24.547429 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27fe6428-6096-49b8-9e99-cd5679f18552 admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:24.557156 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27fe6428-6096-49b8-9e99-cd5679f18552 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:24.557156 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27fe6428-6096-49b8-9e99-cd5679f18552 admin admin] PATH_INFO: `/v3/users` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:24.557156 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27fe6428-6096-49b8-9e99-cd5679f18552 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:24.561114 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27fe6428-6096-49b8-9e99-cd5679f18552 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:24.587245 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1339/2712] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 01:02:24 2022] POST /identity/v3/users => generated 288 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 01:02:24.591304 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaa5397a-4022-4557-900b-4d76bd825d7f admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:24.602145 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaa5397a-4022-4557-900b-4d76bd825d7f admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:24.606874 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64f051c6-2b51-498f-bd65-0ff564748b6a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:24.607474 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64f051c6-2b51-498f-bd65-0ff564748b6a admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:24.607781 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64f051c6-2b51-498f-bd65-0ff564748b6a admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:24.608085 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64f051c6-2b51-498f-bd65-0ff564748b6a admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:24.608678 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64f051c6-2b51-498f-bd65-0ff564748b6a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:24.610250 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64f051c6-2b51-498f-bd65-0ff564748b6a admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:24.625716 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1374/2713] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 01:02:24 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 01:02:24.629582 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27fe6428-6096-49b8-9e99-cd5679f18552 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:24.639623 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27fe6428-6096-49b8-9e99-cd5679f18552 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:24.644484 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56986534-6811-4d79-97f7-3d92a11feebc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:24.645085 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56986534-6811-4d79-97f7-3d92a11feebc admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:24.645366 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56986534-6811-4d79-97f7-3d92a11feebc admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:24.645641 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56986534-6811-4d79-97f7-3d92a11feebc admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:24.646279 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56986534-6811-4d79-97f7-3d92a11feebc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:24.647885 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56986534-6811-4d79-97f7-3d92a11feebc admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:24.664310 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1340/2714] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 01:02:24 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 01:02:24.671322 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64f051c6-2b51-498f-bd65-0ff564748b6a admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:24.683309 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64f051c6-2b51-498f-bd65-0ff564748b6a admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:24.687942 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72458bfe-ddff-4db2-9af4-e6d08ead694f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:24.688592 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72458bfe-ddff-4db2-9af4-e6d08ead694f admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:24.688873 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72458bfe-ddff-4db2-9af4-e6d08ead694f admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:24.689146 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72458bfe-ddff-4db2-9af4-e6d08ead694f admin admin] PATH_INFO: `/v3/projects/fd5c9701cead4c6aa7f6701847268df2/users/448465ca36584280938ccaba5f449d71/roles/588016933bab436d8b7c3ac76827356d` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:24.706370 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72458bfe-ddff-4db2-9af4-e6d08ead694f admin admin] RBAC: Authorizing `identity:create_grant(project_id=fd5c9701cead4c6aa7f6701847268df2, user_id=448465ca36584280938ccaba5f449d71, role_id=588016933bab436d8b7c3ac76827356d)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:24.709091 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72458bfe-ddff-4db2-9af4-e6d08ead694f admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:24.722900 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1375/2715] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 01:02:24 2022] PUT /identity/v3/projects/fd5c9701cead4c6aa7f6701847268df2/users/448465ca36584280938ccaba5f449d71/roles/588016933bab436d8b7c3ac76827356d => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:02:24.733857 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50421231-64f4-42ab-8d42-f55f5d7e0693 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:24.734141 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50421231-64f4-42ab-8d42-f55f5d7e0693 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:24.734417 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50421231-64f4-42ab-8d42-f55f5d7e0693 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:24.761720 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-50421231-64f4-42ab-8d42-f55f5d7e0693 None None] MFA Rules not processed for user `448465ca36584280938ccaba5f449d71`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:02:24.762849 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-50421231-64f4-42ab-8d42-f55f5d7e0693 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:24.779141 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:DELETE=5,keystone:SELECT=91,keystone:INSERT=4 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:02:24.829722 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1341/2716] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:02:24 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:02:24.841973 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72458bfe-ddff-4db2-9af4-e6d08ead694f admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:24.843661 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-72458bfe-ddff-4db2-9af4-e6d08ead694f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:24.877375 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:02:24.902941 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72458bfe-ddff-4db2-9af4-e6d08ead694f admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:24.906626 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1ee9a64-c30f-4057-bcae-d7ca50971ae3 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1f7a88094d73471787cc0a287adbd394', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:24.907251 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1ee9a64-c30f-4057-bcae-d7ca50971ae3 service glance] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:24.907531 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1ee9a64-c30f-4057-bcae-d7ca50971ae3 service glance] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:24.907838 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1ee9a64-c30f-4057-bcae-d7ca50971ae3 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:24.916665 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1ee9a64-c30f-4057-bcae-d7ca50971ae3 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:24.918086 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1ee9a64-c30f-4057-bcae-d7ca50971ae3 service glance] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:24.926756 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1376/2717] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 01:02:24 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 01:02:28.569246 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50421231-64f4-42ab-8d42-f55f5d7e0693 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:28.570670 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-50421231-64f4-42ab-8d42-f55f5d7e0693 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:28.632550 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50421231-64f4-42ab-8d42-f55f5d7e0693 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:28.636128 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d7aa2b0-e5fb-4fd5-9d7a-26e3be55facb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3f48335668a4e8f934ac11ff66cd308', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:28.636722 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d7aa2b0-e5fb-4fd5-9d7a-26e3be55facb service cinder] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:28.637003 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d7aa2b0-e5fb-4fd5-9d7a-26e3be55facb service cinder] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:28.637269 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d7aa2b0-e5fb-4fd5-9d7a-26e3be55facb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:28.638954 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-2d7aa2b0-e5fb-4fd5-9d7a-26e3be55facb service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:28.698898 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d7aa2b0-e5fb-4fd5-9d7a-26e3be55facb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:28.700350 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d7aa2b0-e5fb-4fd5-9d7a-26e3be55facb service cinder] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:28.709601 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1342/2718] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 01:02:28 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 141 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 01:02:33.591750 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1ee9a64-c30f-4057-bcae-d7ca50971ae3 service glance] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:33.593154 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-d1ee9a64-c30f-4057-bcae-d7ca50971ae3 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:33.661240 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1ee9a64-c30f-4057-bcae-d7ca50971ae3 service glance] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:33.665972 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb25dae-db85-4ee4-ae4a-3072d49175c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:33.666590 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb25dae-db85-4ee4-ae4a-3072d49175c1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:33.666871 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb25dae-db85-4ee4-ae4a-3072d49175c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:33.667147 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb25dae-db85-4ee4-ae4a-3072d49175c1 admin admin] PATH_INFO: `/v3/users/5c28bc0f5b33408ba69a74712524f8d9` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:33.668252 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fb25dae-db85-4ee4-ae4a-3072d49175c1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5c28bc0f5b33408ba69a74712524f8d9)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:33.670165 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fb25dae-db85-4ee4-ae4a-3072d49175c1 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:33.717616 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-8fb25dae-db85-4ee4-ae4a-3072d49175c1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5c28bc0f5b33408ba69a74712524f8d9'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:33.724571 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-8fb25dae-db85-4ee4-ae4a-3072d49175c1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5c28bc0f5b33408ba69a74712524f8d9'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:33.725225 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-8fb25dae-db85-4ee4-ae4a-3072d49175c1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5c28bc0f5b33408ba69a74712524f8d9'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:33.725824 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-8fb25dae-db85-4ee4-ae4a-3072d49175c1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5c28bc0f5b33408ba69a74712524f8d9'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:33.733573 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-8fb25dae-db85-4ee4-ae4a-3072d49175c1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5c28bc0f5b33408ba69a74712524f8d9'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:33.751950 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1377/2719] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 01:02:33 2022] DELETE /identity/v3/users/5c28bc0f5b33408ba69a74712524f8d9 => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:02:33.756991 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee381f5e-06dd-4c5e-a3b2-eacd2eab58bd None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:33.757488 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee381f5e-06dd-4c5e-a3b2-eacd2eab58bd None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:33.757681 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee381f5e-06dd-4c5e-a3b2-eacd2eab58bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:33.787095 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-ee381f5e-06dd-4c5e-a3b2-eacd2eab58bd None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:02:33.788543 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-ee381f5e-06dd-4c5e-a3b2-eacd2eab58bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:33.858979 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1343/2720] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:02:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:02:33.866322 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb25dae-db85-4ee4-ae4a-3072d49175c1 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:33.867946 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-8fb25dae-db85-4ee4-ae4a-3072d49175c1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:33.931033 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb25dae-db85-4ee4-ae4a-3072d49175c1 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:33.935224 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbfa65d2-83e0-48fe-aa29-a78e858e95ca service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:33.935904 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbfa65d2-83e0-48fe-aa29-a78e858e95ca service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:33.936247 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbfa65d2-83e0-48fe-aa29-a78e858e95ca service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:33.936568 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbfa65d2-83e0-48fe-aa29-a78e858e95ca service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:33.943915 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:02:33.952177 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbfa65d2-83e0-48fe-aa29-a78e858e95ca service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:33.953715 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbfa65d2-83e0-48fe-aa29-a78e858e95ca service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:33.964830 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1378/2721] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 01:02:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 01:02:34.276657 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee381f5e-06dd-4c5e-a3b2-eacd2eab58bd None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:34.278116 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-ee381f5e-06dd-4c5e-a3b2-eacd2eab58bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:34.290023 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee381f5e-06dd-4c5e-a3b2-eacd2eab58bd None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:34.295144 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eec3a1db-b65e-4921-8e60-07a9c73dd786 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:34.295885 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec3a1db-b65e-4921-8e60-07a9c73dd786 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:34.296161 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec3a1db-b65e-4921-8e60-07a9c73dd786 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:34.296432 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec3a1db-b65e-4921-8e60-07a9c73dd786 admin admin] PATH_INFO: `/v3/projects/6b037d8bc6e240f7a981facfd03c639d` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:34.297550 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eec3a1db-b65e-4921-8e60-07a9c73dd786 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6b037d8bc6e240f7a981facfd03c639d)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:34.299447 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eec3a1db-b65e-4921-8e60-07a9c73dd786 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:34.302659 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:02:34.348602 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-eec3a1db-b65e-4921-8e60-07a9c73dd786 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6b037d8bc6e240f7a981facfd03c639d', 'request_id': 'req-eec3a1db-b65e-4921-8e60-07a9c73dd786'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:34.354093 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-eec3a1db-b65e-4921-8e60-07a9c73dd786 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6b037d8bc6e240f7a981facfd03c639d', 'request_id': 'req-eec3a1db-b65e-4921-8e60-07a9c73dd786'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:34.360438 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-eec3a1db-b65e-4921-8e60-07a9c73dd786 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6b037d8bc6e240f7a981facfd03c639d', 'request_id': 'req-eec3a1db-b65e-4921-8e60-07a9c73dd786'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:34.373156 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-eec3a1db-b65e-4921-8e60-07a9c73dd786 admin admin] The token cache is being invalidate because project 6b037d8bc6e240f7a981facfd03c639d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81453) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 01:02:34.373437 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-eec3a1db-b65e-4921-8e60-07a9c73dd786 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:34.374051 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-eec3a1db-b65e-4921-8e60-07a9c73dd786 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:34.375482 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1344/2722] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 01:02:34 2022] DELETE /identity/v3/projects/6b037d8bc6e240f7a981facfd03c639d => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:02:34.380040 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbfa65d2-83e0-48fe-aa29-a78e858e95ca service neutron] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:34.389599 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbfa65d2-83e0-48fe-aa29-a78e858e95ca service neutron] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:34.445734 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4de19176-3961-413b-828b-a4397ea611b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:34.446352 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de19176-3961-413b-828b-a4397ea611b2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:34.446632 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de19176-3961-413b-828b-a4397ea611b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:34.446905 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de19176-3961-413b-828b-a4397ea611b2 admin admin] PATH_INFO: `/v3/users/4cd6a2e4d308476d9c69fd5a95cb2ae3` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:34.448039 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4de19176-3961-413b-828b-a4397ea611b2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4cd6a2e4d308476d9c69fd5a95cb2ae3)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:34.449933 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4de19176-3961-413b-828b-a4397ea611b2 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:34.494875 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-4de19176-3961-413b-828b-a4397ea611b2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4cd6a2e4d308476d9c69fd5a95cb2ae3'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:34.502366 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-4de19176-3961-413b-828b-a4397ea611b2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4cd6a2e4d308476d9c69fd5a95cb2ae3'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:34.503016 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-4de19176-3961-413b-828b-a4397ea611b2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4cd6a2e4d308476d9c69fd5a95cb2ae3'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:34.504196 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-4de19176-3961-413b-828b-a4397ea611b2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4cd6a2e4d308476d9c69fd5a95cb2ae3'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:34.511154 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-4de19176-3961-413b-828b-a4397ea611b2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4cd6a2e4d308476d9c69fd5a95cb2ae3'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:34.528890 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1379/2723] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 01:02:34 2022] DELETE /identity/v3/users/4cd6a2e4d308476d9c69fd5a95cb2ae3 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:02:34.795516 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eec3a1db-b65e-4921-8e60-07a9c73dd786 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:34.797028 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-eec3a1db-b65e-4921-8e60-07a9c73dd786 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:34.812900 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eec3a1db-b65e-4921-8e60-07a9c73dd786 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:34.872869 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a434713-0059-4441-b157-4256974f7438 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:34.873552 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a434713-0059-4441-b157-4256974f7438 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:34.873830 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a434713-0059-4441-b157-4256974f7438 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:34.874099 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a434713-0059-4441-b157-4256974f7438 admin admin] PATH_INFO: `/v3/projects/ff44ced9f1f241ad8b78680960b60f01` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:34.875728 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a434713-0059-4441-b157-4256974f7438 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ff44ced9f1f241ad8b78680960b60f01)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:34.877697 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a434713-0059-4441-b157-4256974f7438 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:34.923624 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-0a434713-0059-4441-b157-4256974f7438 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ff44ced9f1f241ad8b78680960b60f01', 'request_id': 'req-0a434713-0059-4441-b157-4256974f7438'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:34.929629 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-0a434713-0059-4441-b157-4256974f7438 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ff44ced9f1f241ad8b78680960b60f01', 'request_id': 'req-0a434713-0059-4441-b157-4256974f7438'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:34.935834 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-0a434713-0059-4441-b157-4256974f7438 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ff44ced9f1f241ad8b78680960b60f01', 'request_id': 'req-0a434713-0059-4441-b157-4256974f7438'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:34.950639 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-0a434713-0059-4441-b157-4256974f7438 admin admin] The token cache is being invalidate because project ff44ced9f1f241ad8b78680960b60f01 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81453) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 01:02:34.950935 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-0a434713-0059-4441-b157-4256974f7438 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:34.951864 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-0a434713-0059-4441-b157-4256974f7438 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:34.953661 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1345/2724] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 01:02:34 2022] DELETE /identity/v3/projects/ff44ced9f1f241ad8b78680960b60f01 => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:02:34.959754 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52cb4319-1459-4697-a656-3efa7aa1d822 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:34.960304 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52cb4319-1459-4697-a656-3efa7aa1d822 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:34.960304 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52cb4319-1459-4697-a656-3efa7aa1d822 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:34.982490 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-52cb4319-1459-4697-a656-3efa7aa1d822 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:02:34.983585 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-52cb4319-1459-4697-a656-3efa7aa1d822 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:35.042295 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1380/2725] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:02:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:02:35.052541 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f312c066-cedf-458b-a144-1212d096bfba None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:35.052854 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f312c066-cedf-458b-a144-1212d096bfba None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:35.053111 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f312c066-cedf-458b-a144-1212d096bfba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:35.073847 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-f312c066-cedf-458b-a144-1212d096bfba None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:02:35.075077 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-f312c066-cedf-458b-a144-1212d096bfba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:35.083380 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1346/2726] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:02:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:02:35.087228 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52cb4319-1459-4697-a656-3efa7aa1d822 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:35.097586 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52cb4319-1459-4697-a656-3efa7aa1d822 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:35.101996 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe2c89f5-dcbb-46a8-a345-b874595313a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:35.102579 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe2c89f5-dcbb-46a8-a345-b874595313a3 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:35.102855 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe2c89f5-dcbb-46a8-a345-b874595313a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:35.103121 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe2c89f5-dcbb-46a8-a345-b874595313a3 admin admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:35.103787 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe2c89f5-dcbb-46a8-a345-b874595313a3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 01:02:35.104084 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe2c89f5-dcbb-46a8-a345-b874595313a3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:35.105654 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe2c89f5-dcbb-46a8-a345-b874595313a3 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:35.117661 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1381/2727] 10.222.0.33 () {58 vars in 1153 bytes} [Sun Oct 9 01:02:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 01:02:35.121931 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f312c066-cedf-458b-a144-1212d096bfba None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:35.130940 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f312c066-cedf-458b-a144-1212d096bfba None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:35.135510 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab342e27-6376-4faf-8371-4c87ff85e518 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:35.136131 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab342e27-6376-4faf-8371-4c87ff85e518 admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:35.136415 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab342e27-6376-4faf-8371-4c87ff85e518 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:35.136687 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab342e27-6376-4faf-8371-4c87ff85e518 admin admin] PATH_INFO: `/v3/projects` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:35.137308 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab342e27-6376-4faf-8371-4c87ff85e518 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:35.139205 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab342e27-6376-4faf-8371-4c87ff85e518 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:35.153840 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:02:35.166976 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1347/2728] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 01:02:35 2022] POST /identity/v3/projects => generated 385 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 01:02:35.171284 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe2c89f5-dcbb-46a8-a345-b874595313a3 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:35.177059 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:02:35.183272 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe2c89f5-dcbb-46a8-a345-b874595313a3 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:35.249991 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82858388-dbcf-4d81-aadf-98b346d41ab4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:35.250657 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82858388-dbcf-4d81-aadf-98b346d41ab4 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:35.250784 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82858388-dbcf-4d81-aadf-98b346d41ab4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:35.251046 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82858388-dbcf-4d81-aadf-98b346d41ab4 admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:35.251776 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82858388-dbcf-4d81-aadf-98b346d41ab4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:35.253711 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82858388-dbcf-4d81-aadf-98b346d41ab4 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:35.273950 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1382/2729] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 01:02:35 2022] POST /identity/v3/users => generated 293 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 01:02:35.278368 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab342e27-6376-4faf-8371-4c87ff85e518 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:35.289070 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab342e27-6376-4faf-8371-4c87ff85e518 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:35.293612 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-454ca1ad-cd5d-42a6-aff1-70ece1b25c5c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:35.294204 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-454ca1ad-cd5d-42a6-aff1-70ece1b25c5c admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:35.294487 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-454ca1ad-cd5d-42a6-aff1-70ece1b25c5c admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:35.294761 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-454ca1ad-cd5d-42a6-aff1-70ece1b25c5c admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:35.295393 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-454ca1ad-cd5d-42a6-aff1-70ece1b25c5c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:35.297018 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-454ca1ad-cd5d-42a6-aff1-70ece1b25c5c admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:35.307318 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1348/2730] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 01:02:35 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 01:02:35.311118 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82858388-dbcf-4d81-aadf-98b346d41ab4 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:35.320843 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82858388-dbcf-4d81-aadf-98b346d41ab4 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:35.325406 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f38e6c1b-cd76-43f2-8ee5-c281d4d14155 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:35.325988 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f38e6c1b-cd76-43f2-8ee5-c281d4d14155 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:35.326267 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f38e6c1b-cd76-43f2-8ee5-c281d4d14155 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:35.326538 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f38e6c1b-cd76-43f2-8ee5-c281d4d14155 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:35.327172 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f38e6c1b-cd76-43f2-8ee5-c281d4d14155 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:35.328779 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f38e6c1b-cd76-43f2-8ee5-c281d4d14155 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:35.339912 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1383/2731] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 01:02:35 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 01:02:35.343872 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-454ca1ad-cd5d-42a6-aff1-70ece1b25c5c admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:35.353774 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-454ca1ad-cd5d-42a6-aff1-70ece1b25c5c admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:35.358346 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b375419f-a8ac-49dd-839d-af53bd0d422f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:35.358979 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b375419f-a8ac-49dd-839d-af53bd0d422f admin admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:35.359257 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b375419f-a8ac-49dd-839d-af53bd0d422f admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:35.359528 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b375419f-a8ac-49dd-839d-af53bd0d422f admin admin] PATH_INFO: `/v3/projects/a343ee9be22e41a990e16ae84cc21d49/users/cefe8687296b4a60b87795df117c5a13/roles/588016933bab436d8b7c3ac76827356d` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:35.374104 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b375419f-a8ac-49dd-839d-af53bd0d422f admin admin] RBAC: Authorizing `identity:create_grant(project_id=a343ee9be22e41a990e16ae84cc21d49, user_id=cefe8687296b4a60b87795df117c5a13, role_id=588016933bab436d8b7c3ac76827356d)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:35.376773 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b375419f-a8ac-49dd-839d-af53bd0d422f admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:35.390551 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1349/2732] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 01:02:35 2022] PUT /identity/v3/projects/a343ee9be22e41a990e16ae84cc21d49/users/cefe8687296b4a60b87795df117c5a13/roles/588016933bab436d8b7c3ac76827356d => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:02:35.402260 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cdc0086-b7d3-4eb7-bce5-3a0a6419f698 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:35.402550 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cdc0086-b7d3-4eb7-bce5-3a0a6419f698 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:35.402830 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cdc0086-b7d3-4eb7-bce5-3a0a6419f698 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:35.429980 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-4cdc0086-b7d3-4eb7-bce5-3a0a6419f698 None None] MFA Rules not processed for user `cefe8687296b4a60b87795df117c5a13`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:02:35.431176 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-4cdc0086-b7d3-4eb7-bce5-3a0a6419f698 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:35.499431 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1384/2733] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:02:35 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:02:35.511231 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b375419f-a8ac-49dd-839d-af53bd0d422f admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:35.512978 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-b375419f-a8ac-49dd-839d-af53bd0d422f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:35.573848 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b375419f-a8ac-49dd-839d-af53bd0d422f admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:35.577481 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee9770d8-a2c6-4328-9a8e-7b1a84508dab service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3f48335668a4e8f934ac11ff66cd308', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:35.578079 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee9770d8-a2c6-4328-9a8e-7b1a84508dab service cinder] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:35.578359 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee9770d8-a2c6-4328-9a8e-7b1a84508dab service cinder] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:35.578632 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee9770d8-a2c6-4328-9a8e-7b1a84508dab service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:35.588227 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee9770d8-a2c6-4328-9a8e-7b1a84508dab service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:35.589634 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee9770d8-a2c6-4328-9a8e-7b1a84508dab service cinder] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:35.598847 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1350/2734] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 01:02:35 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 01:02:35.620119 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cdc0086-b7d3-4eb7-bce5-3a0a6419f698 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:35.681858 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cdc0086-b7d3-4eb7-bce5-3a0a6419f698 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:35.686429 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ad3e4c-0ef2-4f4c-9ad1-d37f2cdf7946 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:35.687047 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ad3e4c-0ef2-4f4c-9ad1-d37f2cdf7946 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:35.687326 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ad3e4c-0ef2-4f4c-9ad1-d37f2cdf7946 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:35.687670 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ad3e4c-0ef2-4f4c-9ad1-d37f2cdf7946 admin admin] PATH_INFO: `/v3/users/cefe8687296b4a60b87795df117c5a13` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:35.688803 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47ad3e4c-0ef2-4f4c-9ad1-d37f2cdf7946 admin admin] RBAC: Authorizing `identity:delete_user(user_id=cefe8687296b4a60b87795df117c5a13)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:35.691259 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47ad3e4c-0ef2-4f4c-9ad1-d37f2cdf7946 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:35.703516 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=96,keystone:INSERT=3,keystone:DELETE=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:02:35.738768 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-47ad3e4c-0ef2-4f4c-9ad1-d37f2cdf7946 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cefe8687296b4a60b87795df117c5a13'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:35.747847 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-47ad3e4c-0ef2-4f4c-9ad1-d37f2cdf7946 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cefe8687296b4a60b87795df117c5a13'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:35.748642 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-47ad3e4c-0ef2-4f4c-9ad1-d37f2cdf7946 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cefe8687296b4a60b87795df117c5a13'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:35.749265 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-47ad3e4c-0ef2-4f4c-9ad1-d37f2cdf7946 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cefe8687296b4a60b87795df117c5a13'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:35.758313 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-47ad3e4c-0ef2-4f4c-9ad1-d37f2cdf7946 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cefe8687296b4a60b87795df117c5a13'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:35.777667 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1385/2735] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 01:02:35 2022] DELETE /identity/v3/users/cefe8687296b4a60b87795df117c5a13 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:02:35.782188 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d651971-d8e7-4116-b96a-f8a073f22f90 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:35.782477 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d651971-d8e7-4116-b96a-f8a073f22f90 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:35.782753 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d651971-d8e7-4116-b96a-f8a073f22f90 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:35.802230 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-5d651971-d8e7-4116-b96a-f8a073f22f90 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:02:35.803456 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-5d651971-d8e7-4116-b96a-f8a073f22f90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:35.861207 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1351/2736] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:02:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:02:35.870852 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ad3e4c-0ef2-4f4c-9ad1-d37f2cdf7946 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:35.872266 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-47ad3e4c-0ef2-4f4c-9ad1-d37f2cdf7946 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:35.945862 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ad3e4c-0ef2-4f4c-9ad1-d37f2cdf7946 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:35.949542 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09087df9-55af-4575-9636-49d48801e9e6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:35.950280 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09087df9-55af-4575-9636-49d48801e9e6 service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:35.950516 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09087df9-55af-4575-9636-49d48801e9e6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:35.950755 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09087df9-55af-4575-9636-49d48801e9e6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:35.961302 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09087df9-55af-4575-9636-49d48801e9e6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:35.962760 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09087df9-55af-4575-9636-49d48801e9e6 service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:35.972870 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1386/2737] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 01:02:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 01:02:36.285701 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d651971-d8e7-4116-b96a-f8a073f22f90 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:36.287205 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-5d651971-d8e7-4116-b96a-f8a073f22f90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:36.307096 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d651971-d8e7-4116-b96a-f8a073f22f90 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:36.321160 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c695e250-4597-4798-a898-414a577d496f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:36.321889 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c695e250-4597-4798-a898-414a577d496f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:36.322241 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c695e250-4597-4798-a898-414a577d496f admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:36.322583 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c695e250-4597-4798-a898-414a577d496f admin admin] PATH_INFO: `/v3/projects/a343ee9be22e41a990e16ae84cc21d49` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:36.323947 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c695e250-4597-4798-a898-414a577d496f admin admin] RBAC: Authorizing `identity:delete_project(project_id=a343ee9be22e41a990e16ae84cc21d49)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:36.325915 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c695e250-4597-4798-a898-414a577d496f admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:36.372769 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-c695e250-4597-4798-a898-414a577d496f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a343ee9be22e41a990e16ae84cc21d49', 'request_id': 'req-c695e250-4597-4798-a898-414a577d496f'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:36.377343 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:INSERT=2,keystone:SELECT=92,keystone:DELETE=6 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:02:36.378748 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-c695e250-4597-4798-a898-414a577d496f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a343ee9be22e41a990e16ae84cc21d49', 'request_id': 'req-c695e250-4597-4798-a898-414a577d496f'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:36.386348 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-c695e250-4597-4798-a898-414a577d496f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a343ee9be22e41a990e16ae84cc21d49', 'request_id': 'req-c695e250-4597-4798-a898-414a577d496f'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:36.398960 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-c695e250-4597-4798-a898-414a577d496f admin admin] The token cache is being invalidate because project a343ee9be22e41a990e16ae84cc21d49 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81453) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 01:02:36.399247 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-c695e250-4597-4798-a898-414a577d496f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:36.399967 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-c695e250-4597-4798-a898-414a577d496f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:36.401743 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1352/2738] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 01:02:36 2022] DELETE /identity/v3/projects/a343ee9be22e41a990e16ae84cc21d49 => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:02:36.406672 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeec4258-7180-4c23-979a-94ea38f22d39 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:36.406961 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeec4258-7180-4c23-979a-94ea38f22d39 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:36.407233 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeec4258-7180-4c23-979a-94ea38f22d39 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:36.426743 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-eeec4258-7180-4c23-979a-94ea38f22d39 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:02:36.427971 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-eeec4258-7180-4c23-979a-94ea38f22d39 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:36.485623 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1387/2739] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:02:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:02:36.493217 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109bf6ee-03a1-4bb6-919b-531f326fcf55 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:36.493497 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109bf6ee-03a1-4bb6-919b-531f326fcf55 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:36.493767 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109bf6ee-03a1-4bb6-919b-531f326fcf55 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:36.513240 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-109bf6ee-03a1-4bb6-919b-531f326fcf55 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:02:36.514431 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-109bf6ee-03a1-4bb6-919b-531f326fcf55 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:36.522639 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1353/2740] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:02:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:02:36.526887 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeec4258-7180-4c23-979a-94ea38f22d39 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:36.536354 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeec4258-7180-4c23-979a-94ea38f22d39 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:36.540891 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b0c5e29-296e-4f9e-b95f-f7340cbc162c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:36.541478 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b0c5e29-296e-4f9e-b95f-f7340cbc162c admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:36.541755 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b0c5e29-296e-4f9e-b95f-f7340cbc162c admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:36.542195 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b0c5e29-296e-4f9e-b95f-f7340cbc162c admin admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:36.542659 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b0c5e29-296e-4f9e-b95f-f7340cbc162c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 01:02:36.542959 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b0c5e29-296e-4f9e-b95f-f7340cbc162c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:36.544582 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b0c5e29-296e-4f9e-b95f-f7340cbc162c admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:36.558125 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1388/2741] 10.222.0.33 () {58 vars in 1153 bytes} [Sun Oct 9 01:02:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 01:02:36.562572 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-109bf6ee-03a1-4bb6-919b-531f326fcf55 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:36.572815 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-109bf6ee-03a1-4bb6-919b-531f326fcf55 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:36.577514 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dee16916-700f-4a20-89e5-fba262d54f9f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:36.578141 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee16916-700f-4a20-89e5-fba262d54f9f admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:36.578423 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee16916-700f-4a20-89e5-fba262d54f9f admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:36.578699 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee16916-700f-4a20-89e5-fba262d54f9f admin admin] PATH_INFO: `/v3/projects` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:36.579348 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dee16916-700f-4a20-89e5-fba262d54f9f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:36.581825 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dee16916-700f-4a20-89e5-fba262d54f9f admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:36.608676 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1354/2742] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 01:02:36 2022] POST /identity/v3/projects => generated 397 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 01:02:36.612994 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b0c5e29-296e-4f9e-b95f-f7340cbc162c admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:36.622886 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b0c5e29-296e-4f9e-b95f-f7340cbc162c admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:36.640218 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:02:36.681557 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33f95710-17c9-45c5-8fcf-cc4ac510b845 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:36.682153 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f95710-17c9-45c5-8fcf-cc4ac510b845 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:36.682432 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f95710-17c9-45c5-8fcf-cc4ac510b845 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:36.682702 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f95710-17c9-45c5-8fcf-cc4ac510b845 admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:36.683316 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33f95710-17c9-45c5-8fcf-cc4ac510b845 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:36.685250 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33f95710-17c9-45c5-8fcf-cc4ac510b845 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:36.706431 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1389/2743] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 01:02:36 2022] POST /identity/v3/users => generated 299 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 01:02:36.710478 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dee16916-700f-4a20-89e5-fba262d54f9f admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:36.720484 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dee16916-700f-4a20-89e5-fba262d54f9f admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:36.725615 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdde2fc5-ed5f-492e-a5f7-1e6f3381708f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:36.726200 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdde2fc5-ed5f-492e-a5f7-1e6f3381708f admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:36.726479 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdde2fc5-ed5f-492e-a5f7-1e6f3381708f admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:36.726751 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdde2fc5-ed5f-492e-a5f7-1e6f3381708f admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:36.727413 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdde2fc5-ed5f-492e-a5f7-1e6f3381708f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:36.729022 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdde2fc5-ed5f-492e-a5f7-1e6f3381708f admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:36.738789 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1355/2744] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 01:02:36 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 01:02:36.742573 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33f95710-17c9-45c5-8fcf-cc4ac510b845 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:36.752137 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33f95710-17c9-45c5-8fcf-cc4ac510b845 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:36.757140 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2028cd21-fb6c-4ca4-b205-6e744e2d879b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:36.757713 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2028cd21-fb6c-4ca4-b205-6e744e2d879b admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:36.757997 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2028cd21-fb6c-4ca4-b205-6e744e2d879b admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:36.758264 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2028cd21-fb6c-4ca4-b205-6e744e2d879b admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:36.758888 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2028cd21-fb6c-4ca4-b205-6e744e2d879b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:36.760485 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2028cd21-fb6c-4ca4-b205-6e744e2d879b admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:36.770774 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1390/2745] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 01:02:36 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 01:02:36.774673 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdde2fc5-ed5f-492e-a5f7-1e6f3381708f admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:36.785169 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdde2fc5-ed5f-492e-a5f7-1e6f3381708f admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:36.789722 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-564f075a-8ee3-4cde-a155-0c4b3d529fbe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:36.790355 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564f075a-8ee3-4cde-a155-0c4b3d529fbe admin admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:36.790635 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564f075a-8ee3-4cde-a155-0c4b3d529fbe admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:36.790900 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564f075a-8ee3-4cde-a155-0c4b3d529fbe admin admin] PATH_INFO: `/v3/projects/e78172ea7902488bb2ab00394644f064/users/d9816a22bf584b48877e8953a6939c60/roles/588016933bab436d8b7c3ac76827356d` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:36.805770 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-564f075a-8ee3-4cde-a155-0c4b3d529fbe admin admin] RBAC: Authorizing `identity:create_grant(project_id=e78172ea7902488bb2ab00394644f064, user_id=d9816a22bf584b48877e8953a6939c60, role_id=588016933bab436d8b7c3ac76827356d)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:36.808569 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-564f075a-8ee3-4cde-a155-0c4b3d529fbe admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:36.824329 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1356/2746] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 01:02:36 2022] PUT /identity/v3/projects/e78172ea7902488bb2ab00394644f064/users/d9816a22bf584b48877e8953a6939c60/roles/588016933bab436d8b7c3ac76827356d => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:02:36.836744 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d107c134-892b-4d12-8951-6f34e1229bd6 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:36.837191 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d107c134-892b-4d12-8951-6f34e1229bd6 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:36.837754 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d107c134-892b-4d12-8951-6f34e1229bd6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:36.865801 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-d107c134-892b-4d12-8951-6f34e1229bd6 None None] MFA Rules not processed for user `d9816a22bf584b48877e8953a6939c60`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:02:36.866968 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-d107c134-892b-4d12-8951-6f34e1229bd6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:36.932256 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1391/2747] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:02:36 2022] POST /identity/v3/auth/tokens => generated 3274 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:02:36.943506 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-564f075a-8ee3-4cde-a155-0c4b3d529fbe admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:36.945363 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-564f075a-8ee3-4cde-a155-0c4b3d529fbe admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:37.003077 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-564f075a-8ee3-4cde-a155-0c4b3d529fbe admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:37.006653 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a914d44f-dba9-4473-8ec6-3daa6e28a96b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3f48335668a4e8f934ac11ff66cd308', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:37.007245 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a914d44f-dba9-4473-8ec6-3daa6e28a96b service cinder] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:37.007518 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a914d44f-dba9-4473-8ec6-3daa6e28a96b service cinder] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:37.007816 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a914d44f-dba9-4473-8ec6-3daa6e28a96b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:37.015921 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a914d44f-dba9-4473-8ec6-3daa6e28a96b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:37.017324 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a914d44f-dba9-4473-8ec6-3daa6e28a96b service cinder] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:37.026594 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1357/2748] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 01:02:36 2022] GET /identity/v3/auth/tokens => generated 3274 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 01:02:45.106061 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d107c134-892b-4d12-8951-6f34e1229bd6 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:45.167412 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d107c134-892b-4d12-8951-6f34e1229bd6 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:45.171931 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-654512b6-9936-46f0-af99-f8efc3ae2820 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:45.172546 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-654512b6-9936-46f0-af99-f8efc3ae2820 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:45.172823 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-654512b6-9936-46f0-af99-f8efc3ae2820 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:45.173099 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-654512b6-9936-46f0-af99-f8efc3ae2820 admin admin] PATH_INFO: `/v3/users/d9816a22bf584b48877e8953a6939c60` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:45.174192 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-654512b6-9936-46f0-af99-f8efc3ae2820 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d9816a22bf584b48877e8953a6939c60)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:45.176630 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-654512b6-9936-46f0-af99-f8efc3ae2820 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:45.208874 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=92,keystone:INSERT=3,keystone:DELETE=5 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:02:45.223772 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-654512b6-9936-46f0-af99-f8efc3ae2820 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd9816a22bf584b48877e8953a6939c60'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:45.230583 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-654512b6-9936-46f0-af99-f8efc3ae2820 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd9816a22bf584b48877e8953a6939c60'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:45.231233 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-654512b6-9936-46f0-af99-f8efc3ae2820 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd9816a22bf584b48877e8953a6939c60'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:45.231907 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-654512b6-9936-46f0-af99-f8efc3ae2820 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd9816a22bf584b48877e8953a6939c60'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:45.239465 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-654512b6-9936-46f0-af99-f8efc3ae2820 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd9816a22bf584b48877e8953a6939c60'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:45.257093 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1392/2749] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 01:02:45 2022] DELETE /identity/v3/users/d9816a22bf584b48877e8953a6939c60 => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:02:45.261522 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b41e38fd-1879-462b-af20-63d351470a15 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:45.262006 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b41e38fd-1879-462b-af20-63d351470a15 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:45.262149 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b41e38fd-1879-462b-af20-63d351470a15 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:45.284123 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-b41e38fd-1879-462b-af20-63d351470a15 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:02:45.285341 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-b41e38fd-1879-462b-af20-63d351470a15 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:45.342438 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1358/2750] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:02:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:02:45.352396 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-654512b6-9936-46f0-af99-f8efc3ae2820 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:45.353807 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-654512b6-9936-46f0-af99-f8efc3ae2820 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:45.426086 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-654512b6-9936-46f0-af99-f8efc3ae2820 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:45.430578 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57198a61-a10f-49d3-b8da-87efb8158324 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:45.431211 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57198a61-a10f-49d3-b8da-87efb8158324 service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:45.431492 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57198a61-a10f-49d3-b8da-87efb8158324 service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:45.431799 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57198a61-a10f-49d3-b8da-87efb8158324 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:45.442757 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57198a61-a10f-49d3-b8da-87efb8158324 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:45.444325 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57198a61-a10f-49d3-b8da-87efb8158324 service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:45.455447 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1393/2751] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 01:02:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 01:02:45.839572 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b41e38fd-1879-462b-af20-63d351470a15 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:45.841144 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-b41e38fd-1879-462b-af20-63d351470a15 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:45.856030 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b41e38fd-1879-462b-af20-63d351470a15 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:45.860694 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c276162c-abd3-4670-b2f0-d5d8f1c1114f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:45.862442 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c276162c-abd3-4670-b2f0-d5d8f1c1114f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:45.862718 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c276162c-abd3-4670-b2f0-d5d8f1c1114f admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:45.863051 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c276162c-abd3-4670-b2f0-d5d8f1c1114f admin admin] PATH_INFO: `/v3/projects/e78172ea7902488bb2ab00394644f064` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:45.864145 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c276162c-abd3-4670-b2f0-d5d8f1c1114f admin admin] RBAC: Authorizing `identity:delete_project(project_id=e78172ea7902488bb2ab00394644f064)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:45.866033 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c276162c-abd3-4670-b2f0-d5d8f1c1114f admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:45.887103 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=96,keystone:INSERT=3,keystone:DELETE=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:02:45.915553 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-c276162c-abd3-4670-b2f0-d5d8f1c1114f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e78172ea7902488bb2ab00394644f064', 'request_id': 'req-c276162c-abd3-4670-b2f0-d5d8f1c1114f'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:45.921053 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-c276162c-abd3-4670-b2f0-d5d8f1c1114f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e78172ea7902488bb2ab00394644f064', 'request_id': 'req-c276162c-abd3-4670-b2f0-d5d8f1c1114f'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:45.927055 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-c276162c-abd3-4670-b2f0-d5d8f1c1114f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e78172ea7902488bb2ab00394644f064', 'request_id': 'req-c276162c-abd3-4670-b2f0-d5d8f1c1114f'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:45.939053 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-c276162c-abd3-4670-b2f0-d5d8f1c1114f admin admin] The token cache is being invalidate because project e78172ea7902488bb2ab00394644f064 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81453) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 01:02:45.939314 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-c276162c-abd3-4670-b2f0-d5d8f1c1114f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:45.939957 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-c276162c-abd3-4670-b2f0-d5d8f1c1114f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:45.941422 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1359/2752] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 01:02:45 2022] DELETE /identity/v3/projects/e78172ea7902488bb2ab00394644f064 => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:02:47.842617 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57198a61-a10f-49d3-b8da-87efb8158324 service neutron] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:47.844132 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-57198a61-a10f-49d3-b8da-87efb8158324 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:47.856530 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57198a61-a10f-49d3-b8da-87efb8158324 service neutron] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:47.913867 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d57c7fde-78f7-4180-b5e5-22ccc3491780 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:47.914493 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d57c7fde-78f7-4180-b5e5-22ccc3491780 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:47.914771 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d57c7fde-78f7-4180-b5e5-22ccc3491780 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:47.915045 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d57c7fde-78f7-4180-b5e5-22ccc3491780 admin admin] PATH_INFO: `/v3/users/448465ca36584280938ccaba5f449d71` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:47.916189 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d57c7fde-78f7-4180-b5e5-22ccc3491780 admin admin] RBAC: Authorizing `identity:delete_user(user_id=448465ca36584280938ccaba5f449d71)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:47.918094 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d57c7fde-78f7-4180-b5e5-22ccc3491780 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:47.963712 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-d57c7fde-78f7-4180-b5e5-22ccc3491780 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '448465ca36584280938ccaba5f449d71'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:47.968230 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:02:47.972738 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-d57c7fde-78f7-4180-b5e5-22ccc3491780 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '448465ca36584280938ccaba5f449d71'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:47.973715 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-d57c7fde-78f7-4180-b5e5-22ccc3491780 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '448465ca36584280938ccaba5f449d71'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:47.974656 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-d57c7fde-78f7-4180-b5e5-22ccc3491780 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '448465ca36584280938ccaba5f449d71'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:47.982389 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-d57c7fde-78f7-4180-b5e5-22ccc3491780 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '448465ca36584280938ccaba5f449d71'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:47.999725 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1394/2753] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 01:02:47 2022] DELETE /identity/v3/users/448465ca36584280938ccaba5f449d71 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:02:48.005513 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05200cc-59eb-4928-8193-7a7b6dc36ed1 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:48.006041 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05200cc-59eb-4928-8193-7a7b6dc36ed1 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:48.006041 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05200cc-59eb-4928-8193-7a7b6dc36ed1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:48.025493 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-b05200cc-59eb-4928-8193-7a7b6dc36ed1 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:02:48.027017 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-b05200cc-59eb-4928-8193-7a7b6dc36ed1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:48.092443 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1360/2754] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:02:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:02:48.102334 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d57c7fde-78f7-4180-b5e5-22ccc3491780 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:48.103791 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-d57c7fde-78f7-4180-b5e5-22ccc3491780 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:48.161898 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d57c7fde-78f7-4180-b5e5-22ccc3491780 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:48.165573 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0022f7c-5781-4343-b96f-db68a6867f4e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:48.166177 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0022f7c-5781-4343-b96f-db68a6867f4e service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:48.166462 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0022f7c-5781-4343-b96f-db68a6867f4e service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:48.166762 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0022f7c-5781-4343-b96f-db68a6867f4e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:48.176565 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0022f7c-5781-4343-b96f-db68a6867f4e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:48.178002 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0022f7c-5781-4343-b96f-db68a6867f4e service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:48.187751 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1395/2755] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 01:02:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 01:02:48.451159 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b05200cc-59eb-4928-8193-7a7b6dc36ed1 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:48.452640 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-b05200cc-59eb-4928-8193-7a7b6dc36ed1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:48.467109 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b05200cc-59eb-4928-8193-7a7b6dc36ed1 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:48.472304 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ff715fb-42c2-4608-a0a4-0c55b3020402 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:48.472933 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ff715fb-42c2-4608-a0a4-0c55b3020402 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:48.473208 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ff715fb-42c2-4608-a0a4-0c55b3020402 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:48.473474 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ff715fb-42c2-4608-a0a4-0c55b3020402 admin admin] PATH_INFO: `/v3/projects/fd5c9701cead4c6aa7f6701847268df2` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:48.474583 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ff715fb-42c2-4608-a0a4-0c55b3020402 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fd5c9701cead4c6aa7f6701847268df2)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:48.476484 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ff715fb-42c2-4608-a0a4-0c55b3020402 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:48.531387 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-3ff715fb-42c2-4608-a0a4-0c55b3020402 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fd5c9701cead4c6aa7f6701847268df2', 'request_id': 'req-3ff715fb-42c2-4608-a0a4-0c55b3020402'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:48.537594 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-3ff715fb-42c2-4608-a0a4-0c55b3020402 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fd5c9701cead4c6aa7f6701847268df2', 'request_id': 'req-3ff715fb-42c2-4608-a0a4-0c55b3020402'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:48.543660 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-3ff715fb-42c2-4608-a0a4-0c55b3020402 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fd5c9701cead4c6aa7f6701847268df2', 'request_id': 'req-3ff715fb-42c2-4608-a0a4-0c55b3020402'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:48.556689 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-3ff715fb-42c2-4608-a0a4-0c55b3020402 admin admin] The token cache is being invalidate because project fd5c9701cead4c6aa7f6701847268df2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81453) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 01:02:48.556972 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-3ff715fb-42c2-4608-a0a4-0c55b3020402 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:48.557693 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-3ff715fb-42c2-4608-a0a4-0c55b3020402 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:48.559155 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1361/2756] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 01:02:48 2022] DELETE /identity/v3/projects/fd5c9701cead4c6aa7f6701847268df2 => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:02:48.564058 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe70d863-7c92-4640-be55-645479dbcc71 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:48.564340 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe70d863-7c92-4640-be55-645479dbcc71 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:48.564612 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe70d863-7c92-4640-be55-645479dbcc71 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:48.583311 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-fe70d863-7c92-4640-be55-645479dbcc71 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:02:48.584553 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-fe70d863-7c92-4640-be55-645479dbcc71 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:48.641583 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1396/2757] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:02:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:02:48.649248 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d25234c9-a8a0-4401-b2c4-ea971d6171a8 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:48.649521 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d25234c9-a8a0-4401-b2c4-ea971d6171a8 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:48.649794 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d25234c9-a8a0-4401-b2c4-ea971d6171a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:48.669925 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-d25234c9-a8a0-4401-b2c4-ea971d6171a8 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:02:48.671119 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-d25234c9-a8a0-4401-b2c4-ea971d6171a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:48.679793 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1362/2758] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:02:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:02:48.683997 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe70d863-7c92-4640-be55-645479dbcc71 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:48.693375 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe70d863-7c92-4640-be55-645479dbcc71 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:48.697809 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24b4af96-66e7-4d6d-a7f7-ec69f6e9500e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:48.698391 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b4af96-66e7-4d6d-a7f7-ec69f6e9500e admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:48.698666 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b4af96-66e7-4d6d-a7f7-ec69f6e9500e admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:48.698934 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b4af96-66e7-4d6d-a7f7-ec69f6e9500e admin admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:48.699560 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24b4af96-66e7-4d6d-a7f7-ec69f6e9500e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 01:02:48.699887 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24b4af96-66e7-4d6d-a7f7-ec69f6e9500e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:48.701452 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24b4af96-66e7-4d6d-a7f7-ec69f6e9500e admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:48.712552 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1397/2759] 10.222.0.33 () {58 vars in 1153 bytes} [Sun Oct 9 01:02:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 01:02:48.716544 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d25234c9-a8a0-4401-b2c4-ea971d6171a8 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:48.726039 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d25234c9-a8a0-4401-b2c4-ea971d6171a8 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:48.730679 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a781f7-908c-4d60-8392-c9fd96cfa987 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:48.731263 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a781f7-908c-4d60-8392-c9fd96cfa987 admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:48.731546 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a781f7-908c-4d60-8392-c9fd96cfa987 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:48.731850 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a781f7-908c-4d60-8392-c9fd96cfa987 admin admin] PATH_INFO: `/v3/projects` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:48.732478 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9a781f7-908c-4d60-8392-c9fd96cfa987 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:48.734354 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9a781f7-908c-4d60-8392-c9fd96cfa987 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:48.761098 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1363/2760] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 01:02:48 2022] POST /identity/v3/projects => generated 373 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 09 01:02:48.765137 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24b4af96-66e7-4d6d-a7f7-ec69f6e9500e admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:48.775041 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24b4af96-66e7-4d6d-a7f7-ec69f6e9500e admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:48.830518 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e62245d-d8df-4531-b247-c022beba16c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:48.831646 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e62245d-d8df-4531-b247-c022beba16c4 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:48.831933 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e62245d-d8df-4531-b247-c022beba16c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:48.832211 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e62245d-d8df-4531-b247-c022beba16c4 admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:48.832838 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e62245d-d8df-4531-b247-c022beba16c4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:48.834758 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e62245d-d8df-4531-b247-c022beba16c4 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:48.842531 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=96,keystone:INSERT=1,keystone:DELETE=3 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:02:48.857320 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1398/2761] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 01:02:48 2022] POST /identity/v3/users => generated 287 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 09 01:02:48.861379 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a781f7-908c-4d60-8392-c9fd96cfa987 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:48.866796 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:02:48.872892 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a781f7-908c-4d60-8392-c9fd96cfa987 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:48.878229 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f096eaed-48dd-4bb3-b896-ba3c6ac93b52 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:48.878799 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f096eaed-48dd-4bb3-b896-ba3c6ac93b52 admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:48.879080 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f096eaed-48dd-4bb3-b896-ba3c6ac93b52 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:48.879350 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f096eaed-48dd-4bb3-b896-ba3c6ac93b52 admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:48.880019 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f096eaed-48dd-4bb3-b896-ba3c6ac93b52 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:48.881625 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f096eaed-48dd-4bb3-b896-ba3c6ac93b52 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:48.891521 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1364/2762] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 01:02:48 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 01:02:48.896294 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e62245d-d8df-4531-b247-c022beba16c4 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:48.906181 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e62245d-d8df-4531-b247-c022beba16c4 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:48.911278 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f034222-cbf2-4cf6-9bde-87ebb15533f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:48.911888 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f034222-cbf2-4cf6-9bde-87ebb15533f8 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:48.912165 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f034222-cbf2-4cf6-9bde-87ebb15533f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:48.912434 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f034222-cbf2-4cf6-9bde-87ebb15533f8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:48.913057 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f034222-cbf2-4cf6-9bde-87ebb15533f8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:48.914630 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f034222-cbf2-4cf6-9bde-87ebb15533f8 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:48.925024 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1399/2763] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 01:02:48 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 01:02:48.929151 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f096eaed-48dd-4bb3-b896-ba3c6ac93b52 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:48.938947 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f096eaed-48dd-4bb3-b896-ba3c6ac93b52 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:48.943565 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-855a23bf-831c-471d-9f4c-3c13384ae970 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:48.944234 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-855a23bf-831c-471d-9f4c-3c13384ae970 admin admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:48.944528 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-855a23bf-831c-471d-9f4c-3c13384ae970 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:48.944791 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-855a23bf-831c-471d-9f4c-3c13384ae970 admin admin] PATH_INFO: `/v3/projects/a5885c2f35074c2ab0e51257836943b4/users/113fce1f0c1945a89848472f7636a7e2/roles/588016933bab436d8b7c3ac76827356d` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:48.959387 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-855a23bf-831c-471d-9f4c-3c13384ae970 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a5885c2f35074c2ab0e51257836943b4, user_id=113fce1f0c1945a89848472f7636a7e2, role_id=588016933bab436d8b7c3ac76827356d)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:48.962053 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-855a23bf-831c-471d-9f4c-3c13384ae970 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:48.975034 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1365/2764] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 01:02:48 2022] PUT /identity/v3/projects/a5885c2f35074c2ab0e51257836943b4/users/113fce1f0c1945a89848472f7636a7e2/roles/588016933bab436d8b7c3ac76827356d => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:02:48.986057 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f9fccf-cf4b-4e7e-b96f-57609957711b None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:48.986340 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f9fccf-cf4b-4e7e-b96f-57609957711b None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:48.986613 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f9fccf-cf4b-4e7e-b96f-57609957711b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:49.013996 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-37f9fccf-cf4b-4e7e-b96f-57609957711b None None] MFA Rules not processed for user `113fce1f0c1945a89848472f7636a7e2`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:02:49.015201 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-37f9fccf-cf4b-4e7e-b96f-57609957711b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:49.082698 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1400/2765] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:02:48 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:02:49.088901 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-855a23bf-831c-471d-9f4c-3c13384ae970 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:49.090633 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-855a23bf-831c-471d-9f4c-3c13384ae970 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:49.146543 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-855a23bf-831c-471d-9f4c-3c13384ae970 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:49.149992 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3b49311-0ce3-410d-89d1-05a1526fa312 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3f48335668a4e8f934ac11ff66cd308', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:49.150587 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b49311-0ce3-410d-89d1-05a1526fa312 service cinder] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:49.151116 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b49311-0ce3-410d-89d1-05a1526fa312 service cinder] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:49.151387 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b49311-0ce3-410d-89d1-05a1526fa312 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:49.160032 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3b49311-0ce3-410d-89d1-05a1526fa312 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:49.161431 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3b49311-0ce3-410d-89d1-05a1526fa312 service cinder] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:49.169940 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1366/2766] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 01:02:49 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 01:02:52.187988 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37f9fccf-cf4b-4e7e-b96f-57609957711b None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:52.249077 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37f9fccf-cf4b-4e7e-b96f-57609957711b None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:52.253558 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b786e73b-7849-4720-860b-954fc9a2eb53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:52.254168 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b786e73b-7849-4720-860b-954fc9a2eb53 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:52.254444 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b786e73b-7849-4720-860b-954fc9a2eb53 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:52.254715 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b786e73b-7849-4720-860b-954fc9a2eb53 admin admin] PATH_INFO: `/v3/users/113fce1f0c1945a89848472f7636a7e2` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:52.255831 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b786e73b-7849-4720-860b-954fc9a2eb53 admin admin] RBAC: Authorizing `identity:delete_user(user_id=113fce1f0c1945a89848472f7636a7e2)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:52.257726 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b786e73b-7849-4720-860b-954fc9a2eb53 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:52.303802 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-b786e73b-7849-4720-860b-954fc9a2eb53 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '113fce1f0c1945a89848472f7636a7e2'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:52.310484 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-b786e73b-7849-4720-860b-954fc9a2eb53 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '113fce1f0c1945a89848472f7636a7e2'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:52.311136 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-b786e73b-7849-4720-860b-954fc9a2eb53 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '113fce1f0c1945a89848472f7636a7e2'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:52.311758 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-b786e73b-7849-4720-860b-954fc9a2eb53 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '113fce1f0c1945a89848472f7636a7e2'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:52.319783 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-b786e73b-7849-4720-860b-954fc9a2eb53 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '113fce1f0c1945a89848472f7636a7e2'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:52.324973 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:INSERT=4,keystone:SELECT=87,keystone:DELETE=9 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:02:52.338735 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1401/2767] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 01:02:52 2022] DELETE /identity/v3/users/113fce1f0c1945a89848472f7636a7e2 => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:02:52.343285 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c154c9-5001-430b-81a7-0da10ff72e10 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:52.343773 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c154c9-5001-430b-81a7-0da10ff72e10 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:52.343871 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c154c9-5001-430b-81a7-0da10ff72e10 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:52.363556 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-45c154c9-5001-430b-81a7-0da10ff72e10 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:02:52.364770 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-45c154c9-5001-430b-81a7-0da10ff72e10 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:52.422127 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1367/2768] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:02:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:02:52.432323 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b786e73b-7849-4720-860b-954fc9a2eb53 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:52.433746 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-b786e73b-7849-4720-860b-954fc9a2eb53 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:52.489927 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b786e73b-7849-4720-860b-954fc9a2eb53 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:52.493473 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fe51d8e-88fd-4202-8dfc-e2238bd9c0f8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:52.494058 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe51d8e-88fd-4202-8dfc-e2238bd9c0f8 service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:52.494331 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe51d8e-88fd-4202-8dfc-e2238bd9c0f8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:52.494597 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe51d8e-88fd-4202-8dfc-e2238bd9c0f8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:52.503250 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fe51d8e-88fd-4202-8dfc-e2238bd9c0f8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:52.504683 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fe51d8e-88fd-4202-8dfc-e2238bd9c0f8 service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:52.513771 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1402/2769] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 01:02:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 01:02:52.779180 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45c154c9-5001-430b-81a7-0da10ff72e10 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:52.780888 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-45c154c9-5001-430b-81a7-0da10ff72e10 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:52.794545 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45c154c9-5001-430b-81a7-0da10ff72e10 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:52.799502 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaf00892-2a0f-439f-beca-e58bdba43280 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:52.802810 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf00892-2a0f-439f-beca-e58bdba43280 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:52.802810 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf00892-2a0f-439f-beca-e58bdba43280 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:52.802810 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf00892-2a0f-439f-beca-e58bdba43280 admin admin] PATH_INFO: `/v3/projects/a5885c2f35074c2ab0e51257836943b4` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:52.802810 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaf00892-2a0f-439f-beca-e58bdba43280 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a5885c2f35074c2ab0e51257836943b4)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:52.804022 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaf00892-2a0f-439f-beca-e58bdba43280 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:52.852736 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-aaf00892-2a0f-439f-beca-e58bdba43280 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a5885c2f35074c2ab0e51257836943b4', 'request_id': 'req-aaf00892-2a0f-439f-beca-e58bdba43280'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:52.858880 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-aaf00892-2a0f-439f-beca-e58bdba43280 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a5885c2f35074c2ab0e51257836943b4', 'request_id': 'req-aaf00892-2a0f-439f-beca-e58bdba43280'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:52.865733 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-aaf00892-2a0f-439f-beca-e58bdba43280 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a5885c2f35074c2ab0e51257836943b4', 'request_id': 'req-aaf00892-2a0f-439f-beca-e58bdba43280'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:52.869047 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=92,keystone:INSERT=2,keystone:DELETE=6 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:02:52.880844 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-aaf00892-2a0f-439f-beca-e58bdba43280 admin admin] The token cache is being invalidate because project a5885c2f35074c2ab0e51257836943b4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81453) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 01:02:52.881132 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-aaf00892-2a0f-439f-beca-e58bdba43280 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:52.881758 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-aaf00892-2a0f-439f-beca-e58bdba43280 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:02:52.883359 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1368/2770] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 01:02:52 2022] DELETE /identity/v3/projects/a5885c2f35074c2ab0e51257836943b4 => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:02:52.888221 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c1f5bd-1bc1-4eb4-955b-de417480dc10 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:52.888508 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c1f5bd-1bc1-4eb4-955b-de417480dc10 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:52.888778 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c1f5bd-1bc1-4eb4-955b-de417480dc10 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:52.908939 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-b9c1f5bd-1bc1-4eb4-955b-de417480dc10 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:02:52.910696 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-b9c1f5bd-1bc1-4eb4-955b-de417480dc10 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:52.969129 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1403/2771] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:02:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 09 01:02:52.976983 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5725ef7-bf40-41c1-a251-0f4dc079f2c6 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:52.977272 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5725ef7-bf40-41c1-a251-0f4dc079f2c6 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:52.977582 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5725ef7-bf40-41c1-a251-0f4dc079f2c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:52.996891 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-d5725ef7-bf40-41c1-a251-0f4dc079f2c6 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:02:52.998091 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-d5725ef7-bf40-41c1-a251-0f4dc079f2c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:53.006445 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1369/2772] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:02:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:02:53.010555 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9c1f5bd-1bc1-4eb4-955b-de417480dc10 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:53.020633 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9c1f5bd-1bc1-4eb4-955b-de417480dc10 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:53.025075 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0601f7a8-f581-455c-bac6-dddd8f00aafa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:53.025655 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0601f7a8-f581-455c-bac6-dddd8f00aafa admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:53.026186 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0601f7a8-f581-455c-bac6-dddd8f00aafa admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:53.026455 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0601f7a8-f581-455c-bac6-dddd8f00aafa admin admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:53.027085 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0601f7a8-f581-455c-bac6-dddd8f00aafa admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 01:02:53.027380 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0601f7a8-f581-455c-bac6-dddd8f00aafa admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:53.028982 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0601f7a8-f581-455c-bac6-dddd8f00aafa admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:53.040479 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1404/2773] 10.222.0.33 () {58 vars in 1153 bytes} [Sun Oct 9 01:02:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 01:02:53.044471 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5725ef7-bf40-41c1-a251-0f4dc079f2c6 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:53.053441 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5725ef7-bf40-41c1-a251-0f4dc079f2c6 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:53.058052 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4863ac84-5466-4b1f-b2f7-ea747e23eff9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:53.058629 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4863ac84-5466-4b1f-b2f7-ea747e23eff9 admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:53.058908 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4863ac84-5466-4b1f-b2f7-ea747e23eff9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:53.059206 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4863ac84-5466-4b1f-b2f7-ea747e23eff9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:53.059867 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4863ac84-5466-4b1f-b2f7-ea747e23eff9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:53.061755 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4863ac84-5466-4b1f-b2f7-ea747e23eff9 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:53.091395 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1370/2774] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 01:02:53 2022] POST /identity/v3/projects => generated 367 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 09 01:02:53.095587 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0601f7a8-f581-455c-bac6-dddd8f00aafa admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:53.105528 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0601f7a8-f581-455c-bac6-dddd8f00aafa admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:53.160957 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37d54b90-1205-4061-a685-71b32b8a4af2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:53.161547 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d54b90-1205-4061-a685-71b32b8a4af2 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:53.161824 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d54b90-1205-4061-a685-71b32b8a4af2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:53.162127 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d54b90-1205-4061-a685-71b32b8a4af2 admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:53.162911 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37d54b90-1205-4061-a685-71b32b8a4af2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:53.164874 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37d54b90-1205-4061-a685-71b32b8a4af2 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:53.185110 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1405/2775] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 01:02:53 2022] POST /identity/v3/users => generated 284 bytes in 90 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 01:02:53.189297 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4863ac84-5466-4b1f-b2f7-ea747e23eff9 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:53.199917 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4863ac84-5466-4b1f-b2f7-ea747e23eff9 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:53.204500 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-589abd76-37df-43fa-ac30-bfd229ed81bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:53.205092 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-589abd76-37df-43fa-ac30-bfd229ed81bf admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:53.205375 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-589abd76-37df-43fa-ac30-bfd229ed81bf admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:53.205649 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-589abd76-37df-43fa-ac30-bfd229ed81bf admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:53.206915 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-589abd76-37df-43fa-ac30-bfd229ed81bf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:53.208580 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-589abd76-37df-43fa-ac30-bfd229ed81bf admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:53.218271 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1371/2776] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 01:02:53 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 01:02:53.222025 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37d54b90-1205-4061-a685-71b32b8a4af2 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:53.231750 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37d54b90-1205-4061-a685-71b32b8a4af2 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:53.236224 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-576d2302-7f28-4647-b66e-9ace181be6e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:53.236800 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-576d2302-7f28-4647-b66e-9ace181be6e5 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:53.237077 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-576d2302-7f28-4647-b66e-9ace181be6e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:53.237348 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-576d2302-7f28-4647-b66e-9ace181be6e5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:53.238021 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-576d2302-7f28-4647-b66e-9ace181be6e5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:53.239655 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-576d2302-7f28-4647-b66e-9ace181be6e5 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:53.240843 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=96,keystone:DELETE=1,keystone:INSERT=3 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:02:53.252042 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1406/2777] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 01:02:53 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 01:02:53.256668 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-589abd76-37df-43fa-ac30-bfd229ed81bf admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:53.266602 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-589abd76-37df-43fa-ac30-bfd229ed81bf admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:53.271854 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdde7941-f4dd-49ef-a8b2-fd0cc8b98049 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:53.272490 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdde7941-f4dd-49ef-a8b2-fd0cc8b98049 admin admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:53.272772 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdde7941-f4dd-49ef-a8b2-fd0cc8b98049 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:53.273047 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdde7941-f4dd-49ef-a8b2-fd0cc8b98049 admin admin] PATH_INFO: `/v3/projects/d1cc477f1f0948158585b2f499b16b14/users/b695183fe03a4409acb80eb7c0147bf6/roles/588016933bab436d8b7c3ac76827356d` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:53.287374 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdde7941-f4dd-49ef-a8b2-fd0cc8b98049 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d1cc477f1f0948158585b2f499b16b14, user_id=b695183fe03a4409acb80eb7c0147bf6, role_id=588016933bab436d8b7c3ac76827356d)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:53.290028 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdde7941-f4dd-49ef-a8b2-fd0cc8b98049 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:53.303059 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1372/2778] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 01:02:53 2022] PUT /identity/v3/projects/d1cc477f1f0948158585b2f499b16b14/users/b695183fe03a4409acb80eb7c0147bf6/roles/588016933bab436d8b7c3ac76827356d => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:02:53.314346 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6062d12-1ffe-4f9f-8a80-fa945e3b32dd None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:53.314630 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6062d12-1ffe-4f9f-8a80-fa945e3b32dd None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:53.314925 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6062d12-1ffe-4f9f-8a80-fa945e3b32dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:53.344736 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-d6062d12-1ffe-4f9f-8a80-fa945e3b32dd None None] MFA Rules not processed for user `b695183fe03a4409acb80eb7c0147bf6`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:02:53.345982 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-d6062d12-1ffe-4f9f-8a80-fa945e3b32dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:53.409202 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1407/2779] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:02:53 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 09 01:02:53.420814 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdde7941-f4dd-49ef-a8b2-fd0cc8b98049 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:53.423083 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-fdde7941-f4dd-49ef-a8b2-fd0cc8b98049 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:53.479151 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdde7941-f4dd-49ef-a8b2-fd0cc8b98049 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:53.482773 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-128face1-80f0-4d04-8fc9-ee45fe86355e service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1f7a88094d73471787cc0a287adbd394', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:53.483368 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128face1-80f0-4d04-8fc9-ee45fe86355e service glance] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:53.483682 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128face1-80f0-4d04-8fc9-ee45fe86355e service glance] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:53.483961 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128face1-80f0-4d04-8fc9-ee45fe86355e service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:53.492491 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-128face1-80f0-4d04-8fc9-ee45fe86355e service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:53.493897 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-128face1-80f0-4d04-8fc9-ee45fe86355e service glance] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:53.502553 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1373/2780] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 01:02:53 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 01:02:54.208438 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-331cf4ba-c54f-47e4-bc26-4937001e8f76 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:54.208994 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-331cf4ba-c54f-47e4-bc26-4937001e8f76 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:54.209461 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-331cf4ba-c54f-47e4-bc26-4937001e8f76 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:54.230209 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-331cf4ba-c54f-47e4-bc26-4937001e8f76 None None] MFA Rules not processed for user `5f3247ddcda54394995d844c776bf43c`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:02:54.231501 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-331cf4ba-c54f-47e4-bc26-4937001e8f76 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:54.290444 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1408/2781] 10.222.0.33 () {58 vars in 1009 bytes} [Sun Oct 9 01:02:54 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:02:54.344383 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-128face1-80f0-4d04-8fc9-ee45fe86355e service glance] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:02:54.346087 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-128face1-80f0-4d04-8fc9-ee45fe86355e service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:02:54.420261 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-128face1-80f0-4d04-8fc9-ee45fe86355e service glance] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:02:54.425467 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f38c8356-d55d-4321-9748-2357c7b4736a service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '70e367e802894b08afd7d756d8d9ecff', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:02:54.426147 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f38c8356-d55d-4321-9748-2357c7b4736a service swift] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:02:54.426352 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f38c8356-d55d-4321-9748-2357c7b4736a service swift] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:02:54.426623 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f38c8356-d55d-4321-9748-2357c7b4736a service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:02:54.435144 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f38c8356-d55d-4321-9748-2357c7b4736a service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:02:54.436733 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f38c8356-d55d-4321-9748-2357c7b4736a service swift] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:02:54.446481 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1374/2782] 10.222.0.33 () {60 vars in 1384 bytes} [Sun Oct 9 01:02:54 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 09 01:03:04.281367 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=59 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:03:04.443160 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:DELETE=1,keystone:SELECT=87,keystone:INSERT=2 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:03:07.855521 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-331cf4ba-c54f-47e4-bc26-4937001e8f76 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:03:07.857038 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-331cf4ba-c54f-47e4-bc26-4937001e8f76 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:03:07.916923 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-331cf4ba-c54f-47e4-bc26-4937001e8f76 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:03:07.920768 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32cbad19-a3eb-4c37-b344-98f2a3a302dd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:03:07.921371 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32cbad19-a3eb-4c37-b344-98f2a3a302dd service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:03:07.921647 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32cbad19-a3eb-4c37-b344-98f2a3a302dd service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:03:07.921919 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32cbad19-a3eb-4c37-b344-98f2a3a302dd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:03:07.923650 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-32cbad19-a3eb-4c37-b344-98f2a3a302dd service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:03:07.933417 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32cbad19-a3eb-4c37-b344-98f2a3a302dd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:03:07.934853 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32cbad19-a3eb-4c37-b344-98f2a3a302dd service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:03:07.943914 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1409/2783] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 01:03:07 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 01:03:08.068580 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eee284d-a348-493a-87c2-00e57102424f None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:03:08.069146 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eee284d-a348-493a-87c2-00e57102424f None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:03:08.069262 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eee284d-a348-493a-87c2-00e57102424f None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:03:08.070620 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1375/2784] 10.222.0.33 () {54 vars in 921 bytes} [Sun Oct 9 01:03:08 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 09 01:03:08.078833 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b2ae3ee-0461-46c7-aaab-7a30a85167a3 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:03:08.079459 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b2ae3ee-0461-46c7-aaab-7a30a85167a3 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:03:08.080220 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b2ae3ee-0461-46c7-aaab-7a30a85167a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:03:08.113745 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-8b2ae3ee-0461-46c7-aaab-7a30a85167a3 None None] MFA Rules not processed for user `e0a5d95969e74157bb2d2a57207e1651`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:03:08.114986 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-8b2ae3ee-0461-46c7-aaab-7a30a85167a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:03:08.197240 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1410/2785] 10.222.0.33 () {58 vars in 1018 bytes} [Sun Oct 9 01:03:08 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:03:08.215878 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eee284d-a348-493a-87c2-00e57102424f None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:03:08.215878 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-0eee284d-a348-493a-87c2-00e57102424f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:03:08.226146 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eee284d-a348-493a-87c2-00e57102424f None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:03:08.229680 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ca2e81d-cde4-49c2-a1c4-845fa160c7bb service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1f7a88094d73471787cc0a287adbd394', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:03:08.230276 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca2e81d-cde4-49c2-a1c4-845fa160c7bb service glance] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:03:08.230684 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca2e81d-cde4-49c2-a1c4-845fa160c7bb service glance] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:03:08.230964 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca2e81d-cde4-49c2-a1c4-845fa160c7bb service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:03:08.242844 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ca2e81d-cde4-49c2-a1c4-845fa160c7bb service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:03:08.244367 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ca2e81d-cde4-49c2-a1c4-845fa160c7bb service glance] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:03:08.255975 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1376/2786] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 01:03:08 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 50 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 01:03:08.330463 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-576f833a-1e5d-4a1d-aaa9-540a17881233 None None] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:03:08.330750 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-576f833a-1e5d-4a1d-aaa9-540a17881233 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:03:08.331018 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-576f833a-1e5d-4a1d-aaa9-540a17881233 None None] PATH_INFO: `/` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:03:08.332386 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1411/2787] 10.222.0.33 () {54 vars in 921 bytes} [Sun Oct 9 01:03:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 01:03:08.337696 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41dcc66b-68a8-4dd4-aa44-d2f41782fa07 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:03:08.338205 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41dcc66b-68a8-4dd4-aa44-d2f41782fa07 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:03:08.338604 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41dcc66b-68a8-4dd4-aa44-d2f41782fa07 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:03:08.359309 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-41dcc66b-68a8-4dd4-aa44-d2f41782fa07 None None] MFA Rules not processed for user `7d2eabf9865847ceb6e44d55126078a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:03:08.360542 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-41dcc66b-68a8-4dd4-aa44-d2f41782fa07 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:03:08.367500 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1377/2788] 10.222.0.33 () {58 vars in 1018 bytes} [Sun Oct 9 01:03:08 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:03:08.379026 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-576f833a-1e5d-4a1d-aaa9-540a17881233 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:03:08.390598 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-576f833a-1e5d-4a1d-aaa9-540a17881233 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:03:08.394167 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bb1b68d-014d-457e-b9bd-576cb078c3ff service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:03:08.394751 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb1b68d-014d-457e-b9bd-576cb078c3ff service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:03:08.395037 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb1b68d-014d-457e-b9bd-576cb078c3ff service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:03:08.395312 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb1b68d-014d-457e-b9bd-576cb078c3ff service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:03:08.402984 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bb1b68d-014d-457e-b9bd-576cb078c3ff service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:03:08.404441 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bb1b68d-014d-457e-b9bd-576cb078c3ff service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:03:08.412114 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1412/2789] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 01:03:08 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 34 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 01:03:08.493949 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-646a483d-5db1-4c31-8f83-b4b3cc317342 None None] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:03:08.494266 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-646a483d-5db1-4c31-8f83-b4b3cc317342 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:03:08.494542 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-646a483d-5db1-4c31-8f83-b4b3cc317342 None None] PATH_INFO: `/` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:03:08.495955 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1378/2790] 10.222.0.33 () {54 vars in 921 bytes} [Sun Oct 9 01:03:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 09 01:03:08.501304 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b85d6e77-4d6a-4bea-b700-bf74720763a5 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:03:08.501665 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b85d6e77-4d6a-4bea-b700-bf74720763a5 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:03:08.502031 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b85d6e77-4d6a-4bea-b700-bf74720763a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:03:08.522759 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-b85d6e77-4d6a-4bea-b700-bf74720763a5 None None] MFA Rules not processed for user `e0a5d95969e74157bb2d2a57207e1651`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:03:08.523976 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-b85d6e77-4d6a-4bea-b700-bf74720763a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:03:08.533455 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1413/2791] 10.222.0.33 () {58 vars in 1018 bytes} [Sun Oct 9 01:03:08 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:03:08.543760 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-646a483d-5db1-4c31-8f83-b4b3cc317342 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:03:08.555400 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-646a483d-5db1-4c31-8f83-b4b3cc317342 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:03:08.558936 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4df4c8d-d0e0-4b4a-94cf-5ba9f9dc4154 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:03:08.559519 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4df4c8d-d0e0-4b4a-94cf-5ba9f9dc4154 service neutron] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:03:08.559831 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4df4c8d-d0e0-4b4a-94cf-5ba9f9dc4154 service neutron] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:03:08.560103 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4df4c8d-d0e0-4b4a-94cf-5ba9f9dc4154 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:03:08.570543 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4df4c8d-d0e0-4b4a-94cf-5ba9f9dc4154 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:03:08.571978 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4df4c8d-d0e0-4b4a-94cf-5ba9f9dc4154 service neutron] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:03:08.582295 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1379/2792] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 01:03:08 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 01:03:15.587738 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b3dd00-edb9-4088-91ff-1d49cf163a2c None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:03:15.588277 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b3dd00-edb9-4088-91ff-1d49cf163a2c None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:03:15.588277 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b3dd00-edb9-4088-91ff-1d49cf163a2c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:03:15.606752 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-08b3dd00-edb9-4088-91ff-1d49cf163a2c None None] MFA Rules not processed for user `5f3247ddcda54394995d844c776bf43c`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:03:15.607950 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-08b3dd00-edb9-4088-91ff-1d49cf163a2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:03:15.614805 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1414/2793] 10.222.0.33 () {58 vars in 1009 bytes} [Sun Oct 9 01:03:15 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 28 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:03:15.626637 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4df4c8d-d0e0-4b4a-94cf-5ba9f9dc4154 service neutron] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:03:15.638235 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4df4c8d-d0e0-4b4a-94cf-5ba9f9dc4154 service neutron] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:03:15.643013 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d260cb15-c663-47bb-8a1f-feba5047162c service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '70e367e802894b08afd7d756d8d9ecff', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:03:15.643686 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d260cb15-c663-47bb-8a1f-feba5047162c service swift] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:03:15.643960 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d260cb15-c663-47bb-8a1f-feba5047162c service swift] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:03:15.644227 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d260cb15-c663-47bb-8a1f-feba5047162c service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:03:15.652898 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d260cb15-c663-47bb-8a1f-feba5047162c service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:03:15.654434 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d260cb15-c663-47bb-8a1f-feba5047162c service swift] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:03:15.662108 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1380/2794] 10.222.0.33 () {60 vars in 1384 bytes} [Sun Oct 9 01:03:15 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 09 01:03:25.199181 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08b3dd00-edb9-4088-91ff-1d49cf163a2c None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:03:25.260324 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08b3dd00-edb9-4088-91ff-1d49cf163a2c None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:03:25.264742 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41d02455-86dc-4694-ab94-286236267178 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:03:25.265360 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41d02455-86dc-4694-ab94-286236267178 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:03:25.265640 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41d02455-86dc-4694-ab94-286236267178 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:03:25.265913 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41d02455-86dc-4694-ab94-286236267178 admin admin] PATH_INFO: `/v3/users/b695183fe03a4409acb80eb7c0147bf6` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:03:25.267023 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41d02455-86dc-4694-ab94-286236267178 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b695183fe03a4409acb80eb7c0147bf6)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:03:25.268955 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41d02455-86dc-4694-ab94-286236267178 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:03:25.279474 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:03:25.315102 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-41d02455-86dc-4694-ab94-286236267178 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b695183fe03a4409acb80eb7c0147bf6'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:03:25.322144 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-41d02455-86dc-4694-ab94-286236267178 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b695183fe03a4409acb80eb7c0147bf6'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:03:25.322802 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-41d02455-86dc-4694-ab94-286236267178 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b695183fe03a4409acb80eb7c0147bf6'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:03:25.323420 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-41d02455-86dc-4694-ab94-286236267178 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b695183fe03a4409acb80eb7c0147bf6'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:03:25.330416 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-41d02455-86dc-4694-ab94-286236267178 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b695183fe03a4409acb80eb7c0147bf6'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:03:25.347350 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1415/2795] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 01:03:25 2022] DELETE /identity/v3/users/b695183fe03a4409acb80eb7c0147bf6 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:03:25.352327 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf6191d-e9ad-4761-b80c-6c57c4b6f911 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:03:25.352867 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf6191d-e9ad-4761-b80c-6c57c4b6f911 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:03:25.352867 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf6191d-e9ad-4761-b80c-6c57c4b6f911 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:03:25.371403 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-bdf6191d-e9ad-4761-b80c-6c57c4b6f911 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:03:25.372618 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-bdf6191d-e9ad-4761-b80c-6c57c4b6f911 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:03:25.429371 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1381/2796] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:03:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:03:25.438860 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41d02455-86dc-4694-ab94-286236267178 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:03:25.440273 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-41d02455-86dc-4694-ab94-286236267178 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:03:25.498239 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41d02455-86dc-4694-ab94-286236267178 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:03:25.501755 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f1c05cf-a35b-4ea8-9c12-b00668a8d0ed service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:03:25.502339 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f1c05cf-a35b-4ea8-9c12-b00668a8d0ed service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:03:25.502615 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f1c05cf-a35b-4ea8-9c12-b00668a8d0ed service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:03:25.502882 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f1c05cf-a35b-4ea8-9c12-b00668a8d0ed service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:03:25.511517 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f1c05cf-a35b-4ea8-9c12-b00668a8d0ed service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:03:25.512950 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f1c05cf-a35b-4ea8-9c12-b00668a8d0ed service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:03:25.522555 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1416/2797] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 01:03:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 01:03:25.797075 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdf6191d-e9ad-4761-b80c-6c57c4b6f911 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:03:25.798511 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-bdf6191d-e9ad-4761-b80c-6c57c4b6f911 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:03:25.810597 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdf6191d-e9ad-4761-b80c-6c57c4b6f911 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:03:25.815220 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1be5e40f-c80f-46db-b199-926e0f1ce377 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:03:25.815880 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be5e40f-c80f-46db-b199-926e0f1ce377 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:03:25.816163 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be5e40f-c80f-46db-b199-926e0f1ce377 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:03:25.816436 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be5e40f-c80f-46db-b199-926e0f1ce377 admin admin] PATH_INFO: `/v3/projects/d1cc477f1f0948158585b2f499b16b14` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:03:25.817562 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1be5e40f-c80f-46db-b199-926e0f1ce377 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d1cc477f1f0948158585b2f499b16b14)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:03:25.819475 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1be5e40f-c80f-46db-b199-926e0f1ce377 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:03:25.865656 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-1be5e40f-c80f-46db-b199-926e0f1ce377 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd1cc477f1f0948158585b2f499b16b14', 'request_id': 'req-1be5e40f-c80f-46db-b199-926e0f1ce377'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:03:25.871084 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-1be5e40f-c80f-46db-b199-926e0f1ce377 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd1cc477f1f0948158585b2f499b16b14', 'request_id': 'req-1be5e40f-c80f-46db-b199-926e0f1ce377'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:03:25.877127 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-1be5e40f-c80f-46db-b199-926e0f1ce377 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd1cc477f1f0948158585b2f499b16b14', 'request_id': 'req-1be5e40f-c80f-46db-b199-926e0f1ce377'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:03:25.889762 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-1be5e40f-c80f-46db-b199-926e0f1ce377 admin admin] The token cache is being invalidate because project d1cc477f1f0948158585b2f499b16b14 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81453) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 01:03:25.890076 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-1be5e40f-c80f-46db-b199-926e0f1ce377 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:03:25.890704 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-1be5e40f-c80f-46db-b199-926e0f1ce377 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:03:25.892228 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1382/2798] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 01:03:25 2022] DELETE /identity/v3/projects/d1cc477f1f0948158585b2f499b16b14 => generated 0 bytes in 96 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:03:25.898145 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fffec19a-42dd-4744-bd0e-2dc551f0aa29 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:03:25.898467 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fffec19a-42dd-4744-bd0e-2dc551f0aa29 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:03:25.898870 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fffec19a-42dd-4744-bd0e-2dc551f0aa29 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:03:25.918320 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-fffec19a-42dd-4744-bd0e-2dc551f0aa29 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:03:25.919490 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-fffec19a-42dd-4744-bd0e-2dc551f0aa29 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:03:25.980981 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1417/2799] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:03:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:03:25.988850 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d927920-b557-40b9-809a-356a2771fe11 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:03:25.989214 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d927920-b557-40b9-809a-356a2771fe11 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:03:25.989489 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d927920-b557-40b9-809a-356a2771fe11 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:03:26.009351 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fffec19a-42dd-4744-bd0e-2dc551f0aa29 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:03:26.010764 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-fffec19a-42dd-4744-bd0e-2dc551f0aa29 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:03:26.012727 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-1d927920-b557-40b9-809a-356a2771fe11 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:03:26.013892 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-1d927920-b557-40b9-809a-356a2771fe11 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:03:26.024005 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1383/2800] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:03:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:03:26.028849 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d927920-b557-40b9-809a-356a2771fe11 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:03:26.039810 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d927920-b557-40b9-809a-356a2771fe11 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:03:26.045287 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35765d49-4e53-4086-b5e1-aa8240c59b37 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:03:26.045884 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35765d49-4e53-4086-b5e1-aa8240c59b37 admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:03:26.046156 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35765d49-4e53-4086-b5e1-aa8240c59b37 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:03:26.046428 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35765d49-4e53-4086-b5e1-aa8240c59b37 admin admin] PATH_INFO: `/v3/domains` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:03:26.047062 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35765d49-4e53-4086-b5e1-aa8240c59b37 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 01:03:26.047356 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35765d49-4e53-4086-b5e1-aa8240c59b37 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:03:26.049045 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35765d49-4e53-4086-b5e1-aa8240c59b37 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:03:26.050886 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:03:26.062462 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1384/2801] 10.222.0.33 () {58 vars in 1153 bytes} [Sun Oct 9 01:03:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 09 01:03:26.067063 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35765d49-4e53-4086-b5e1-aa8240c59b37 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:03:26.079405 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fffec19a-42dd-4744-bd0e-2dc551f0aa29 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:03:26.081097 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35765d49-4e53-4086-b5e1-aa8240c59b37 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:03:26.084352 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d56254-c3bd-4738-92d9-95f03c8266e7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3f48335668a4e8f934ac11ff66cd308', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:03:26.084963 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d56254-c3bd-4738-92d9-95f03c8266e7 service cinder] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:03:26.085371 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d56254-c3bd-4738-92d9-95f03c8266e7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:03:26.085676 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d56254-c3bd-4738-92d9-95f03c8266e7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:03:26.087448 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-19d56254-c3bd-4738-92d9-95f03c8266e7 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:03:26.091667 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f6f6614-ec28-4ad1-8a84-6d317fe51829 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:03:26.092114 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f6f6614-ec28-4ad1-8a84-6d317fe51829 admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:03:26.092390 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f6f6614-ec28-4ad1-8a84-6d317fe51829 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:03:26.093228 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f6f6614-ec28-4ad1-8a84-6d317fe51829 admin admin] PATH_INFO: `/v3/projects` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:03:26.093863 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f6f6614-ec28-4ad1-8a84-6d317fe51829 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:03:26.095791 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f6f6614-ec28-4ad1-8a84-6d317fe51829 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:03:26.115882 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:03:26.131351 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1385/2802] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 01:03:26 2022] POST /identity/v3/projects => generated 383 bytes in 64 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 09 01:03:26.135174 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f6f6614-ec28-4ad1-8a84-6d317fe51829 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:03:26.163751 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19d56254-c3bd-4738-92d9-95f03c8266e7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:03:26.165196 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19d56254-c3bd-4738-92d9-95f03c8266e7 service cinder] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:03:26.174273 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1418/2803] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 01:03:26 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 165 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 09 01:03:26.200836 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f6f6614-ec28-4ad1-8a84-6d317fe51829 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:03:26.205813 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e07b61c9-9ad0-475a-82f9-098289121278 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:03:26.206419 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e07b61c9-9ad0-475a-82f9-098289121278 admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:03:26.206701 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e07b61c9-9ad0-475a-82f9-098289121278 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:03:26.206976 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e07b61c9-9ad0-475a-82f9-098289121278 admin admin] PATH_INFO: `/v3/users` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:03:26.207626 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e07b61c9-9ad0-475a-82f9-098289121278 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:03:26.210131 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e07b61c9-9ad0-475a-82f9-098289121278 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:03:26.230881 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1386/2804] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 01:03:26 2022] POST /identity/v3/users => generated 292 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 09 01:03:26.235080 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d56254-c3bd-4738-92d9-95f03c8266e7 service cinder] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:03:26.246379 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d56254-c3bd-4738-92d9-95f03c8266e7 service cinder] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:03:26.251775 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec6a0b58-9c41-4bc7-addc-4bef6b0aa52f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:03:26.252369 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec6a0b58-9c41-4bc7-addc-4bef6b0aa52f admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:03:26.252647 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec6a0b58-9c41-4bc7-addc-4bef6b0aa52f admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:03:26.252946 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec6a0b58-9c41-4bc7-addc-4bef6b0aa52f admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:03:26.253578 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec6a0b58-9c41-4bc7-addc-4bef6b0aa52f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:03:26.255783 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec6a0b58-9c41-4bc7-addc-4bef6b0aa52f admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:03:26.267957 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1419/2805] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 01:03:26 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 01:03:26.272033 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e07b61c9-9ad0-475a-82f9-098289121278 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:03:26.283159 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e07b61c9-9ad0-475a-82f9-098289121278 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:03:26.288018 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9744ecf0-2844-4dd2-b15b-1f94f071568f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:03:26.288606 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9744ecf0-2844-4dd2-b15b-1f94f071568f admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:03:26.288881 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9744ecf0-2844-4dd2-b15b-1f94f071568f admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:03:26.289150 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9744ecf0-2844-4dd2-b15b-1f94f071568f admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:03:26.289779 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9744ecf0-2844-4dd2-b15b-1f94f071568f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:03:26.291487 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9744ecf0-2844-4dd2-b15b-1f94f071568f admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:03:26.302763 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1387/2806] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 01:03:26 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 01:03:26.307249 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec6a0b58-9c41-4bc7-addc-4bef6b0aa52f admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:03:26.319461 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec6a0b58-9c41-4bc7-addc-4bef6b0aa52f admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:03:26.324410 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef6fb0b8-8779-40da-b0a0-9f96eb609f31 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:03:26.325062 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef6fb0b8-8779-40da-b0a0-9f96eb609f31 admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:03:26.325342 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef6fb0b8-8779-40da-b0a0-9f96eb609f31 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:03:26.325615 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef6fb0b8-8779-40da-b0a0-9f96eb609f31 admin admin] PATH_INFO: `/v3/projects/f37de57a22a246f0935f85392898dd42/users/3650b0b2b7014e0f8a7a5af9ce92374a/roles/588016933bab436d8b7c3ac76827356d` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:03:26.344028 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef6fb0b8-8779-40da-b0a0-9f96eb609f31 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f37de57a22a246f0935f85392898dd42, user_id=3650b0b2b7014e0f8a7a5af9ce92374a, role_id=588016933bab436d8b7c3ac76827356d)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:03:26.347195 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef6fb0b8-8779-40da-b0a0-9f96eb609f31 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:03:26.362097 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1420/2807] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 01:03:26 2022] PUT /identity/v3/projects/f37de57a22a246f0935f85392898dd42/users/3650b0b2b7014e0f8a7a5af9ce92374a/roles/588016933bab436d8b7c3ac76827356d => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:03:26.368218 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7404d610-dc1b-41fd-b479-2e8cd166d515 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:03:26.368704 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7404d610-dc1b-41fd-b479-2e8cd166d515 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:03:26.368704 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7404d610-dc1b-41fd-b479-2e8cd166d515 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:03:26.390444 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-7404d610-dc1b-41fd-b479-2e8cd166d515 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:03:26.391736 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-7404d610-dc1b-41fd-b479-2e8cd166d515 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:03:26.456834 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1388/2808] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:03:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:03:26.466368 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef6fb0b8-8779-40da-b0a0-9f96eb609f31 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:03:26.469250 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-ef6fb0b8-8779-40da-b0a0-9f96eb609f31 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:03:26.534766 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef6fb0b8-8779-40da-b0a0-9f96eb609f31 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:03:26.538693 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a90908f-bda9-43cc-8225-54d2a7396d3c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:03:26.539322 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a90908f-bda9-43cc-8225-54d2a7396d3c service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:03:26.539612 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a90908f-bda9-43cc-8225-54d2a7396d3c service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:03:26.539906 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a90908f-bda9-43cc-8225-54d2a7396d3c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:03:26.549206 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a90908f-bda9-43cc-8225-54d2a7396d3c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:03:26.550641 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a90908f-bda9-43cc-8225-54d2a7396d3c service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:03:26.562123 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1421/2809] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 01:03:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 01:03:30.543391 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3111ef75-3d53-450a-a471-647221441416 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:03:30.543952 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3111ef75-3d53-450a-a471-647221441416 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:03:30.544055 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3111ef75-3d53-450a-a471-647221441416 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:03:30.572447 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-3111ef75-3d53-450a-a471-647221441416 None None] MFA Rules not processed for user `3650b0b2b7014e0f8a7a5af9ce92374a`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:03:30.573599 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-3111ef75-3d53-450a-a471-647221441416 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:03:30.651396 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1389/2810] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:03:30 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 09 01:03:30.656047 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a90908f-bda9-43cc-8225-54d2a7396d3c service neutron] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:03:30.666355 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a90908f-bda9-43cc-8225-54d2a7396d3c service neutron] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:03:30.670974 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b75b53f7-48d1-4076-8d37-a59cb3ddedbf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:03:30.671633 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b75b53f7-48d1-4076-8d37-a59cb3ddedbf admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:03:30.671917 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b75b53f7-48d1-4076-8d37-a59cb3ddedbf admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:03:30.672192 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b75b53f7-48d1-4076-8d37-a59cb3ddedbf admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:03:30.672832 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b75b53f7-48d1-4076-8d37-a59cb3ddedbf admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:03:30.674722 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b75b53f7-48d1-4076-8d37-a59cb3ddedbf admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:03:30.702614 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1422/2811] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 01:03:30 2022] POST /identity/v3/projects => generated 385 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 09 01:03:30.706416 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3111ef75-3d53-450a-a471-647221441416 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:03:30.713103 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:03:30.718039 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3111ef75-3d53-450a-a471-647221441416 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:03:30.797499 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e063f52e-a3f5-41fa-86cd-fce3a4674f37 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:03:30.798101 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e063f52e-a3f5-41fa-86cd-fce3a4674f37 admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:03:30.798389 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e063f52e-a3f5-41fa-86cd-fce3a4674f37 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:03:30.798662 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e063f52e-a3f5-41fa-86cd-fce3a4674f37 admin admin] PATH_INFO: `/v3/users` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:03:30.799293 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e063f52e-a3f5-41fa-86cd-fce3a4674f37 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:03:30.804297 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e063f52e-a3f5-41fa-86cd-fce3a4674f37 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:03:30.825013 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1390/2812] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 01:03:30 2022] POST /identity/v3/users => generated 299 bytes in 119 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 01:03:30.829096 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b75b53f7-48d1-4076-8d37-a59cb3ddedbf admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:03:30.842488 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b75b53f7-48d1-4076-8d37-a59cb3ddedbf admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:03:30.847986 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f1dce1a-b5ea-4791-ac9f-eb8be2a07d8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:03:30.849821 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f1dce1a-b5ea-4791-ac9f-eb8be2a07d8e admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:03:30.850109 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f1dce1a-b5ea-4791-ac9f-eb8be2a07d8e admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:03:30.850379 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f1dce1a-b5ea-4791-ac9f-eb8be2a07d8e admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:03:30.851029 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f1dce1a-b5ea-4791-ac9f-eb8be2a07d8e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:03:30.852675 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f1dce1a-b5ea-4791-ac9f-eb8be2a07d8e admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:03:30.864695 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1423/2813] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 01:03:30 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 01:03:30.868523 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e063f52e-a3f5-41fa-86cd-fce3a4674f37 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:03:30.879822 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e063f52e-a3f5-41fa-86cd-fce3a4674f37 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:03:30.889422 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd28f666-ecbc-492b-9b8f-ec271b5cdf00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:03:30.890329 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd28f666-ecbc-492b-9b8f-ec271b5cdf00 admin admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:03:30.890514 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd28f666-ecbc-492b-9b8f-ec271b5cdf00 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:03:30.890791 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd28f666-ecbc-492b-9b8f-ec271b5cdf00 admin admin] PATH_INFO: `/v3/projects/7e5245a0acb348b8a6764ca11ac5b746/users/3a26861111784509bb0d232b26418396/roles/092036ee46ac427baf4d56468575f697` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:03:30.910379 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd28f666-ecbc-492b-9b8f-ec271b5cdf00 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7e5245a0acb348b8a6764ca11ac5b746, user_id=3a26861111784509bb0d232b26418396, role_id=092036ee46ac427baf4d56468575f697)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:03:30.913795 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd28f666-ecbc-492b-9b8f-ec271b5cdf00 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:03:30.933801 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1391/2814] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 01:03:30 2022] PUT /identity/v3/projects/7e5245a0acb348b8a6764ca11ac5b746/users/3a26861111784509bb0d232b26418396/roles/092036ee46ac427baf4d56468575f697 => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:03:30.943237 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f1dce1a-b5ea-4791-ac9f-eb8be2a07d8e admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:03:30.955961 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f1dce1a-b5ea-4791-ac9f-eb8be2a07d8e admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:03:30.979995 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:03:31.037657 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a0f58cb-3bc5-4409-b82a-f3359a39612b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:03:31.038612 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a0f58cb-3bc5-4409-b82a-f3359a39612b admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:03:31.039181 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a0f58cb-3bc5-4409-b82a-f3359a39612b admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:03:31.040384 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a0f58cb-3bc5-4409-b82a-f3359a39612b admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:03:31.041250 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a0f58cb-3bc5-4409-b82a-f3359a39612b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:03:31.043555 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a0f58cb-3bc5-4409-b82a-f3359a39612b admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:03:31.056675 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1424/2815] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 01:03:30 2022] GET /identity/v3/roles => generated 1420 bytes in 114 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 01:03:31.064273 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd28f666-ecbc-492b-9b8f-ec271b5cdf00 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:03:31.094079 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd28f666-ecbc-492b-9b8f-ec271b5cdf00 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:03:31.099372 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5597575f-f014-41c6-8b43-e453073bf257 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:03:31.100072 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5597575f-f014-41c6-8b43-e453073bf257 admin admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:03:31.100351 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5597575f-f014-41c6-8b43-e453073bf257 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:03:31.100694 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5597575f-f014-41c6-8b43-e453073bf257 admin admin] PATH_INFO: `/v3/projects/7e5245a0acb348b8a6764ca11ac5b746/users/3a26861111784509bb0d232b26418396/roles/092036ee46ac427baf4d56468575f697` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:03:31.102654 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5597575f-f014-41c6-8b43-e453073bf257 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7e5245a0acb348b8a6764ca11ac5b746, user_id=3a26861111784509bb0d232b26418396, role_id=092036ee46ac427baf4d56468575f697)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:03:31.105669 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5597575f-f014-41c6-8b43-e453073bf257 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:03:31.117861 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1392/2816] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 01:03:31 2022] PUT /identity/v3/projects/7e5245a0acb348b8a6764ca11ac5b746/users/3a26861111784509bb0d232b26418396/roles/092036ee46ac427baf4d56468575f697 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:03:32.236903 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ef0462-1d17-47da-9d5b-95b3cc8aaea9 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:03:32.238110 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ef0462-1d17-47da-9d5b-95b3cc8aaea9 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:03:32.238387 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ef0462-1d17-47da-9d5b-95b3cc8aaea9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:03:32.265151 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-25ef0462-1d17-47da-9d5b-95b3cc8aaea9 None None] MFA Rules not processed for user `04bc22d558c2466ba43a456d08bdd6bd`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:03:32.266384 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-25ef0462-1d17-47da-9d5b-95b3cc8aaea9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:03:32.355245 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1425/2817] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:03:32 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:03:36.536946 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e16d22c-463a-4a91-93d1-6d86dcd367bf None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:03:36.537797 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e16d22c-463a-4a91-93d1-6d86dcd367bf None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:03:36.538438 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e16d22c-463a-4a91-93d1-6d86dcd367bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:03:36.594520 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-1e16d22c-463a-4a91-93d1-6d86dcd367bf None None] MFA Rules not processed for user `3a26861111784509bb0d232b26418396`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:03:36.595931 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-1e16d22c-463a-4a91-93d1-6d86dcd367bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:03:36.706528 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1393/2818] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:03:36 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:03:36.720169 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25ef0462-1d17-47da-9d5b-95b3cc8aaea9 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:03:36.722739 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-25ef0462-1d17-47da-9d5b-95b3cc8aaea9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:03:36.797926 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25ef0462-1d17-47da-9d5b-95b3cc8aaea9 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:03:36.802189 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8856cf3c-73d7-40d1-8f2c-1af2b3203bed service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1f7a88094d73471787cc0a287adbd394', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:03:36.802959 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8856cf3c-73d7-40d1-8f2c-1af2b3203bed service glance] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:03:36.803241 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8856cf3c-73d7-40d1-8f2c-1af2b3203bed service glance] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:03:36.803512 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8856cf3c-73d7-40d1-8f2c-1af2b3203bed service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:03:36.869700 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8856cf3c-73d7-40d1-8f2c-1af2b3203bed service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:03:36.871534 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8856cf3c-73d7-40d1-8f2c-1af2b3203bed service glance] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:03:36.881074 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1426/2819] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 01:03:36 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 161 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 01:03:37.085569 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857ab2f4-e797-4c44-8dde-ca010f75c8e9 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:03:37.085569 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857ab2f4-e797-4c44-8dde-ca010f75c8e9 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:03:37.085569 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857ab2f4-e797-4c44-8dde-ca010f75c8e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:03:37.109823 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-857ab2f4-e797-4c44-8dde-ca010f75c8e9 None None] MFA Rules not processed for user `5f3247ddcda54394995d844c776bf43c`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:03:37.111167 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-857ab2f4-e797-4c44-8dde-ca010f75c8e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:03:37.157439 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:03:37.204524 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1394/2820] 10.222.0.33 () {58 vars in 1009 bytes} [Sun Oct 9 01:03:37 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:03:37.219394 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8856cf3c-73d7-40d1-8f2c-1af2b3203bed service glance] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:03:37.221211 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-8856cf3c-73d7-40d1-8f2c-1af2b3203bed service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:03:37.256870 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:03:37.353883 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8856cf3c-73d7-40d1-8f2c-1af2b3203bed service glance] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:03:37.359583 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24a2026c-aaf9-4ead-95df-48e0bcb4b9fe service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '70e367e802894b08afd7d756d8d9ecff', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:03:37.359996 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a2026c-aaf9-4ead-95df-48e0bcb4b9fe service swift] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:03:37.360561 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a2026c-aaf9-4ead-95df-48e0bcb4b9fe service swift] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:03:37.360976 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a2026c-aaf9-4ead-95df-48e0bcb4b9fe service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:03:37.372763 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24a2026c-aaf9-4ead-95df-48e0bcb4b9fe service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:03:37.374305 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24a2026c-aaf9-4ead-95df-48e0bcb4b9fe service swift] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:03:37.387324 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1427/2821] 10.222.0.33 () {60 vars in 1384 bytes} [Sun Oct 9 01:03:37 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 168 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 09 01:03:37.764043 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f894a45b-d4a4-4cf1-8ab7-0c5b40c51126 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:03:37.764545 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f894a45b-d4a4-4cf1-8ab7-0c5b40c51126 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:03:37.764798 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f894a45b-d4a4-4cf1-8ab7-0c5b40c51126 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:03:37.822747 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-f894a45b-d4a4-4cf1-8ab7-0c5b40c51126 None None] MFA Rules not processed for user `5f3247ddcda54394995d844c776bf43c`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:03:37.824279 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-f894a45b-d4a4-4cf1-8ab7-0c5b40c51126 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:03:37.842102 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1395/2822] 10.222.0.33 () {58 vars in 967 bytes} [Sun Oct 9 01:03:37 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:03:37.882445 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24a2026c-aaf9-4ead-95df-48e0bcb4b9fe service swift] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:03:37.902652 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24a2026c-aaf9-4ead-95df-48e0bcb4b9fe service swift] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:03:37.910931 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5946021f-dd7e-479a-ad84-e1822f37222a service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '70e367e802894b08afd7d756d8d9ecff', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:03:37.911722 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5946021f-dd7e-479a-ad84-e1822f37222a service swift] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:03:37.912307 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5946021f-dd7e-479a-ad84-e1822f37222a service swift] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:03:37.913158 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5946021f-dd7e-479a-ad84-e1822f37222a service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:03:37.929578 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5946021f-dd7e-479a-ad84-e1822f37222a service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:03:37.931525 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5946021f-dd7e-479a-ad84-e1822f37222a service swift] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:03:37.943535 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1428/2823] 10.222.0.33 () {60 vars in 1384 bytes} [Sun Oct 9 01:03:37 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 64 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 09 01:03:46.963932 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-defc01b2-e06f-47bf-ba8f-ff6115efa8c4 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:03:46.964510 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-defc01b2-e06f-47bf-ba8f-ff6115efa8c4 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:03:46.964510 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-defc01b2-e06f-47bf-ba8f-ff6115efa8c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:03:46.983853 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-defc01b2-e06f-47bf-ba8f-ff6115efa8c4 None None] MFA Rules not processed for user `3650b0b2b7014e0f8a7a5af9ce92374a`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:03:46.985065 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-defc01b2-e06f-47bf-ba8f-ff6115efa8c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:03:46.991990 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1396/2824] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:03:46 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:03:47.938746 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=31 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:03:48.983442 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e725c25-a287-4284-9ab7-df6bea07337c None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:03:48.984696 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e725c25-a287-4284-9ab7-df6bea07337c None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:03:48.985915 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e725c25-a287-4284-9ab7-df6bea07337c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:03:49.007945 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-9e725c25-a287-4284-9ab7-df6bea07337c None None] MFA Rules not processed for user `5f3247ddcda54394995d844c776bf43c`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:03:49.009190 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-9e725c25-a287-4284-9ab7-df6bea07337c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:03:49.016509 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1429/2825] 10.222.0.33 () {58 vars in 1009 bytes} [Sun Oct 9 01:03:48 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 09 01:03:49.030618 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-defc01b2-e06f-47bf-ba8f-ff6115efa8c4 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:03:49.045060 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-defc01b2-e06f-47bf-ba8f-ff6115efa8c4 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:03:49.051518 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c6113e-944a-4444-863e-fffb2fd5232c service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '70e367e802894b08afd7d756d8d9ecff', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:03:49.052559 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c6113e-944a-4444-863e-fffb2fd5232c service swift] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:03:49.053031 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c6113e-944a-4444-863e-fffb2fd5232c service swift] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:03:49.053664 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c6113e-944a-4444-863e-fffb2fd5232c service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:03:49.062851 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2c6113e-944a-4444-863e-fffb2fd5232c service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:03:49.064485 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2c6113e-944a-4444-863e-fffb2fd5232c service swift] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:03:49.073669 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1397/2826] 10.222.0.33 () {60 vars in 1384 bytes} [Sun Oct 9 01:03:49 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 09 01:03:59.001900 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=3 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:03:59.069551 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=18 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:04:19.284121 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e725c25-a287-4284-9ab7-df6bea07337c None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:04:19.366400 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e725c25-a287-4284-9ab7-df6bea07337c None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:04:19.370012 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25386994-8081-484d-a556-d828535fffa0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:04:19.370630 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25386994-8081-484d-a556-d828535fffa0 service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:04:19.370908 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25386994-8081-484d-a556-d828535fffa0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:04:19.371175 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25386994-8081-484d-a556-d828535fffa0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:04:19.400004 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25386994-8081-484d-a556-d828535fffa0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:04:19.401661 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25386994-8081-484d-a556-d828535fffa0 service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:04:19.413353 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1430/2827] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 01:04:19 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 130 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 01:04:24.219810 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c6113e-944a-4444-863e-fffb2fd5232c service swift] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:04:24.221440 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-c2c6113e-944a-4444-863e-fffb2fd5232c service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:04:24.372353 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c6113e-944a-4444-863e-fffb2fd5232c service swift] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:04:24.377668 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1222494-c42f-410a-b4f0-2ed67fe65b0b service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd25e240cb93643ed8122ccac9586936f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:04:24.378266 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1222494-c42f-410a-b4f0-2ed67fe65b0b service placement] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:04:24.378544 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1222494-c42f-410a-b4f0-2ed67fe65b0b service placement] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:04:24.379351 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1222494-c42f-410a-b4f0-2ed67fe65b0b service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:04:24.381107 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-d1222494-c42f-410a-b4f0-2ed67fe65b0b service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:04:24.392571 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1222494-c42f-410a-b4f0-2ed67fe65b0b service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:04:24.394323 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1222494-c42f-410a-b4f0-2ed67fe65b0b service placement] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:04:24.405315 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1398/2828] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 01:04:24 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 186 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 01:04:27.259641 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0d02c0a-bad0-49f4-bcfe-ea8f3b8976b8 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:04:27.260202 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0d02c0a-bad0-49f4-bcfe-ea8f3b8976b8 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:04:27.260202 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0d02c0a-bad0-49f4-bcfe-ea8f3b8976b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:04:27.279820 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-f0d02c0a-bad0-49f4-bcfe-ea8f3b8976b8 None None] MFA Rules not processed for user `56b5d764f0044358ac594a2edd6f4c0f`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:04:27.281003 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-f0d02c0a-bad0-49f4-bcfe-ea8f3b8976b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:04:27.334737 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1431/2829] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:04:27 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 76 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:04:34.400727 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=59 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:04:37.327377 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=73 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:05:36.542916 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f69ed9f-752c-4a11-9227-07bd20d00495 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:05:36.543419 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f69ed9f-752c-4a11-9227-07bd20d00495 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:05:36.543729 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f69ed9f-752c-4a11-9227-07bd20d00495 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:05:36.574832 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-0f69ed9f-752c-4a11-9227-07bd20d00495 None None] MFA Rules not processed for user `5f3247ddcda54394995d844c776bf43c`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:05:36.576119 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-0f69ed9f-752c-4a11-9227-07bd20d00495 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:05:36.590528 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1399/2830] 10.222.0.33 () {58 vars in 1009 bytes} [Sun Oct 9 01:05:36 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 49 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:05:36.631516 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0d02c0a-bad0-49f4-bcfe-ea8f3b8976b8 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:05:36.644233 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0d02c0a-bad0-49f4-bcfe-ea8f3b8976b8 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:05:36.649142 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c2225bb-776c-4149-b399-634dabcd3934 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '70e367e802894b08afd7d756d8d9ecff', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:05:36.650269 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c2225bb-776c-4149-b399-634dabcd3934 service swift] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:05:36.650552 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c2225bb-776c-4149-b399-634dabcd3934 service swift] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:05:36.650846 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c2225bb-776c-4149-b399-634dabcd3934 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:05:36.659216 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c2225bb-776c-4149-b399-634dabcd3934 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:05:36.660882 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c2225bb-776c-4149-b399-634dabcd3934 service swift] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:05:36.669448 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1432/2831] 10.222.0.33 () {60 vars in 1384 bytes} [Sun Oct 9 01:05:36 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 09 01:05:46.584640 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=10 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:05:46.666384 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=6 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:05:48.412238 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc3e600-a750-4007-994f-394ad52bd873 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:05:48.412756 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc3e600-a750-4007-994f-394ad52bd873 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:05:48.412756 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc3e600-a750-4007-994f-394ad52bd873 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:05:48.432131 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-3cc3e600-a750-4007-994f-394ad52bd873 None None] MFA Rules not processed for user `5f3247ddcda54394995d844c776bf43c`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:05:48.433313 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-3cc3e600-a750-4007-994f-394ad52bd873 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:05:48.440223 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1400/2832] 10.222.0.33 () {58 vars in 967 bytes} [Sun Oct 9 01:05:48 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:05:48.453711 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c2225bb-776c-4149-b399-634dabcd3934 service swift] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:05:48.464773 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c2225bb-776c-4149-b399-634dabcd3934 service swift] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:05:48.469661 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9680b106-b5ba-434c-9b69-9b6e985bcd02 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '70e367e802894b08afd7d756d8d9ecff', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:05:48.470239 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9680b106-b5ba-434c-9b69-9b6e985bcd02 service swift] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:05:48.470513 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9680b106-b5ba-434c-9b69-9b6e985bcd02 service swift] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:05:48.470779 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9680b106-b5ba-434c-9b69-9b6e985bcd02 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:05:48.478522 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9680b106-b5ba-434c-9b69-9b6e985bcd02 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:05:48.480115 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9680b106-b5ba-434c-9b69-9b6e985bcd02 service swift] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:05:48.487844 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1433/2833] 10.222.0.33 () {60 vars in 1384 bytes} [Sun Oct 9 01:05:48 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 35 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 09 01:05:52.067238 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cc3e600-a750-4007-994f-394ad52bd873 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:05:52.072691 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-3cc3e600-a750-4007-994f-394ad52bd873 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:05:52.100397 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cc3e600-a750-4007-994f-394ad52bd873 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:05:52.105985 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f47dce9e-74fe-4cc7-9023-e76648a6bd97 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd25e240cb93643ed8122ccac9586936f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:05:52.107580 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f47dce9e-74fe-4cc7-9023-e76648a6bd97 service placement] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:05:52.108587 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f47dce9e-74fe-4cc7-9023-e76648a6bd97 service placement] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:05:52.109242 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f47dce9e-74fe-4cc7-9023-e76648a6bd97 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:05:52.111704 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-f47dce9e-74fe-4cc7-9023-e76648a6bd97 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:05:52.125445 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f47dce9e-74fe-4cc7-9023-e76648a6bd97 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:05:52.127059 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f47dce9e-74fe-4cc7-9023-e76648a6bd97 service placement] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:05:52.139673 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1401/2834] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 01:05:52 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 73 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 01:05:54.172860 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9680b106-b5ba-434c-9b69-9b6e985bcd02 service swift] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:05:54.187204 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9680b106-b5ba-434c-9b69-9b6e985bcd02 service swift] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:05:54.197702 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-682c0a25-6f32-49a8-a7ab-d68f136f45e8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:05:54.198577 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-682c0a25-6f32-49a8-a7ab-d68f136f45e8 service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:05:54.199007 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-682c0a25-6f32-49a8-a7ab-d68f136f45e8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:05:54.199293 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-682c0a25-6f32-49a8-a7ab-d68f136f45e8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:05:54.201316 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-682c0a25-6f32-49a8-a7ab-d68f136f45e8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:05:54.304883 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-682c0a25-6f32-49a8-a7ab-d68f136f45e8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:05:54.306618 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-682c0a25-6f32-49a8-a7ab-d68f136f45e8 service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:05:54.330209 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1434/2835] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 01:05:54 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 158 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 01:05:55.225967 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f47dce9e-74fe-4cc7-9023-e76648a6bd97 service placement] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:05:55.227923 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-f47dce9e-74fe-4cc7-9023-e76648a6bd97 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:05:55.251950 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f47dce9e-74fe-4cc7-9023-e76648a6bd97 service placement] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:05:55.256044 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05d12870-f0fc-43f0-8394-5f3e6a9c805f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:05:55.256638 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d12870-f0fc-43f0-8394-5f3e6a9c805f service neutron] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:05:55.256913 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d12870-f0fc-43f0-8394-5f3e6a9c805f service neutron] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:05:55.257180 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d12870-f0fc-43f0-8394-5f3e6a9c805f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:05:55.259559 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-05d12870-f0fc-43f0-8394-5f3e6a9c805f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:05:55.275013 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05d12870-f0fc-43f0-8394-5f3e6a9c805f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:05:55.276499 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05d12870-f0fc-43f0-8394-5f3e6a9c805f service neutron] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:05:55.289022 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1402/2836] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 01:05:55 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 64 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 09 01:05:56.488934 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-682c0a25-6f32-49a8-a7ab-d68f136f45e8 service neutron] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:05:56.608025 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-682c0a25-6f32-49a8-a7ab-d68f136f45e8 service neutron] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:05:56.618820 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cdcfb9e-5b75-4a58-88fe-375cf5e2d877 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:05:56.619532 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cdcfb9e-5b75-4a58-88fe-375cf5e2d877 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:05:56.620000 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cdcfb9e-5b75-4a58-88fe-375cf5e2d877 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:05:56.620445 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cdcfb9e-5b75-4a58-88fe-375cf5e2d877 admin admin] PATH_INFO: `/v3/users/3650b0b2b7014e0f8a7a5af9ce92374a` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:05:56.622805 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cdcfb9e-5b75-4a58-88fe-375cf5e2d877 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3650b0b2b7014e0f8a7a5af9ce92374a)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:05:56.626232 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cdcfb9e-5b75-4a58-88fe-375cf5e2d877 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:05:56.671507 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:05:56.685264 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-6cdcfb9e-5b75-4a58-88fe-375cf5e2d877 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3650b0b2b7014e0f8a7a5af9ce92374a'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:05:56.693113 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-6cdcfb9e-5b75-4a58-88fe-375cf5e2d877 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3650b0b2b7014e0f8a7a5af9ce92374a'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:05:56.693803 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-6cdcfb9e-5b75-4a58-88fe-375cf5e2d877 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3650b0b2b7014e0f8a7a5af9ce92374a'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:05:56.694415 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-6cdcfb9e-5b75-4a58-88fe-375cf5e2d877 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3650b0b2b7014e0f8a7a5af9ce92374a'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:05:56.702539 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-6cdcfb9e-5b75-4a58-88fe-375cf5e2d877 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3650b0b2b7014e0f8a7a5af9ce92374a'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:05:56.720993 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1435/2837] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 01:05:56 2022] DELETE /identity/v3/users/3650b0b2b7014e0f8a7a5af9ce92374a => generated 0 bytes in 233 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:05:56.842757 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca83817-3ef4-4021-bce1-b3ddd3e24c4b None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:05:56.843246 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca83817-3ef4-4021-bce1-b3ddd3e24c4b None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:05:56.843367 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca83817-3ef4-4021-bce1-b3ddd3e24c4b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:05:56.863257 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-6ca83817-3ef4-4021-bce1-b3ddd3e24c4b None None] MFA Rules not processed for user `5f3247ddcda54394995d844c776bf43c`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:05:56.864508 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-6ca83817-3ef4-4021-bce1-b3ddd3e24c4b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:05:56.913279 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cdcfb9e-5b75-4a58-88fe-375cf5e2d877 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:05:56.917542 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-6cdcfb9e-5b75-4a58-88fe-375cf5e2d877 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:05:56.936415 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1403/2838] 10.222.0.33 () {58 vars in 1009 bytes} [Sun Oct 9 01:05:56 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 09 01:05:56.951145 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ca83817-3ef4-4021-bce1-b3ddd3e24c4b None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:05:56.953186 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-6ca83817-3ef4-4021-bce1-b3ddd3e24c4b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:05:57.062816 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ca83817-3ef4-4021-bce1-b3ddd3e24c4b None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:05:57.069926 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32e893eb-e1c8-41fc-849c-4d364fa266ff service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '70e367e802894b08afd7d756d8d9ecff', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:05:57.071175 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e893eb-e1c8-41fc-849c-4d364fa266ff service swift] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:05:57.071433 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e893eb-e1c8-41fc-849c-4d364fa266ff service swift] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:05:57.071722 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e893eb-e1c8-41fc-849c-4d364fa266ff service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:05:57.081127 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cdcfb9e-5b75-4a58-88fe-375cf5e2d877 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:05:57.082802 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32e893eb-e1c8-41fc-849c-4d364fa266ff service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:05:57.088219 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfb60d1a-b459-4019-bd3a-f1819ff80849 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:05:57.088426 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32e893eb-e1c8-41fc-849c-4d364fa266ff service swift] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:05:57.092183 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb60d1a-b459-4019-bd3a-f1819ff80849 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:05:57.092467 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb60d1a-b459-4019-bd3a-f1819ff80849 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:05:57.092741 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb60d1a-b459-4019-bd3a-f1819ff80849 admin admin] PATH_INFO: `/v3/projects/f37de57a22a246f0935f85392898dd42` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:05:57.093961 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfb60d1a-b459-4019-bd3a-f1819ff80849 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f37de57a22a246f0935f85392898dd42)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:05:57.095918 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfb60d1a-b459-4019-bd3a-f1819ff80849 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:05:57.099897 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1404/2839] 10.222.0.33 () {60 vars in 1384 bytes} [Sun Oct 9 01:05:56 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 149 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 09 01:05:57.193013 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-dfb60d1a-b459-4019-bd3a-f1819ff80849 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f37de57a22a246f0935f85392898dd42', 'request_id': 'req-dfb60d1a-b459-4019-bd3a-f1819ff80849'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:05:57.209304 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-dfb60d1a-b459-4019-bd3a-f1819ff80849 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f37de57a22a246f0935f85392898dd42', 'request_id': 'req-dfb60d1a-b459-4019-bd3a-f1819ff80849'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:05:57.226340 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-dfb60d1a-b459-4019-bd3a-f1819ff80849 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f37de57a22a246f0935f85392898dd42', 'request_id': 'req-dfb60d1a-b459-4019-bd3a-f1819ff80849'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:05:57.254188 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-dfb60d1a-b459-4019-bd3a-f1819ff80849 admin admin] The token cache is being invalidate because project f37de57a22a246f0935f85392898dd42 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 01:05:57.254930 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-dfb60d1a-b459-4019-bd3a-f1819ff80849 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:05:57.256399 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-dfb60d1a-b459-4019-bd3a-f1819ff80849 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:05:57.259744 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1436/2840] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 01:05:56 2022] DELETE /identity/v3/projects/f37de57a22a246f0935f85392898dd42 => generated 0 bytes in 347 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:05:57.263504 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32e893eb-e1c8-41fc-849c-4d364fa266ff service swift] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:05:57.275074 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32e893eb-e1c8-41fc-849c-4d364fa266ff service swift] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:05:57.379524 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:05:57.405932 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b910a9a6-a3bb-4de7-897d-a516d291118e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:05:57.406633 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b910a9a6-a3bb-4de7-897d-a516d291118e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:05:57.407038 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b910a9a6-a3bb-4de7-897d-a516d291118e admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:05:57.407322 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b910a9a6-a3bb-4de7-897d-a516d291118e admin admin] PATH_INFO: `/v3/users/3a26861111784509bb0d232b26418396` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:05:57.412478 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b910a9a6-a3bb-4de7-897d-a516d291118e admin admin] RBAC: Authorizing `identity:delete_user(user_id=3a26861111784509bb0d232b26418396)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:05:57.420575 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b910a9a6-a3bb-4de7-897d-a516d291118e admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:05:57.494835 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-b910a9a6-a3bb-4de7-897d-a516d291118e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3a26861111784509bb0d232b26418396'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:05:57.514052 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-b910a9a6-a3bb-4de7-897d-a516d291118e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3a26861111784509bb0d232b26418396'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:05:57.515816 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-b910a9a6-a3bb-4de7-897d-a516d291118e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3a26861111784509bb0d232b26418396'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:05:57.528334 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-b910a9a6-a3bb-4de7-897d-a516d291118e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3a26861111784509bb0d232b26418396'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:05:57.550763 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-b910a9a6-a3bb-4de7-897d-a516d291118e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3a26861111784509bb0d232b26418396'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:05:57.559529 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1405/2841] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 01:05:57 2022] DELETE /identity/v3/users/3a26861111784509bb0d232b26418396 => generated 0 bytes in 296 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:05:57.799137 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfb60d1a-b459-4019-bd3a-f1819ff80849 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:05:57.800786 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-dfb60d1a-b459-4019-bd3a-f1819ff80849 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:05:57.852422 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfb60d1a-b459-4019-bd3a-f1819ff80849 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:05:57.900844 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:05:57.934984 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b4b26a9-284a-40e8-a4a9-98f34a5b1111 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:05:57.935689 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b4b26a9-284a-40e8-a4a9-98f34a5b1111 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:05:57.935933 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b4b26a9-284a-40e8-a4a9-98f34a5b1111 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:05:57.936241 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b4b26a9-284a-40e8-a4a9-98f34a5b1111 admin admin] PATH_INFO: `/v3/projects/7e5245a0acb348b8a6764ca11ac5b746` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:05:57.937348 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b4b26a9-284a-40e8-a4a9-98f34a5b1111 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7e5245a0acb348b8a6764ca11ac5b746)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:05:57.939427 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b4b26a9-284a-40e8-a4a9-98f34a5b1111 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:05:57.993771 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-5b4b26a9-284a-40e8-a4a9-98f34a5b1111 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7e5245a0acb348b8a6764ca11ac5b746', 'request_id': 'req-5b4b26a9-284a-40e8-a4a9-98f34a5b1111'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:05:58.000226 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-5b4b26a9-284a-40e8-a4a9-98f34a5b1111 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7e5245a0acb348b8a6764ca11ac5b746', 'request_id': 'req-5b4b26a9-284a-40e8-a4a9-98f34a5b1111'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:05:58.006728 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-5b4b26a9-284a-40e8-a4a9-98f34a5b1111 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7e5245a0acb348b8a6764ca11ac5b746', 'request_id': 'req-5b4b26a9-284a-40e8-a4a9-98f34a5b1111'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:05:58.026185 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-5b4b26a9-284a-40e8-a4a9-98f34a5b1111 admin admin] The token cache is being invalidate because project 7e5245a0acb348b8a6764ca11ac5b746 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 01:05:58.026626 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-5b4b26a9-284a-40e8-a4a9-98f34a5b1111 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:05:58.027266 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-5b4b26a9-284a-40e8-a4a9-98f34a5b1111 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:05:58.029040 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1437/2842] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 01:05:57 2022] DELETE /identity/v3/projects/7e5245a0acb348b8a6764ca11ac5b746 => generated 0 bytes in 230 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:06:07.554810 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=30,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:06:08.022379 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=36,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:06:08.682949 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b910a9a6-a3bb-4de7-897d-a516d291118e admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:06:08.684439 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-b910a9a6-a3bb-4de7-897d-a516d291118e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:06:08.740116 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b910a9a6-a3bb-4de7-897d-a516d291118e admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:06:08.786868 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85952664-7fbc-43c6-bdf7-5d2af17c674d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:06:08.787467 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85952664-7fbc-43c6-bdf7-5d2af17c674d service neutron] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:06:08.787781 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85952664-7fbc-43c6-bdf7-5d2af17c674d service neutron] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:06:08.788056 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85952664-7fbc-43c6-bdf7-5d2af17c674d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:06:08.790296 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-85952664-7fbc-43c6-bdf7-5d2af17c674d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:06:08.850728 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85952664-7fbc-43c6-bdf7-5d2af17c674d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:06:08.852818 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85952664-7fbc-43c6-bdf7-5d2af17c674d service neutron] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:06:08.862343 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1406/2843] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 01:06:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 180 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 01:06:10.753159 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c12386a-0d24-40b4-af9f-e7cc732ff8d8 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:06:10.753785 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c12386a-0d24-40b4-af9f-e7cc732ff8d8 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:06:10.753785 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c12386a-0d24-40b4-af9f-e7cc732ff8d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:06:10.807713 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-4c12386a-0d24-40b4-af9f-e7cc732ff8d8 None None] MFA Rules not processed for user `5f3247ddcda54394995d844c776bf43c`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:06:10.808926 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-4c12386a-0d24-40b4-af9f-e7cc732ff8d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:06:10.878239 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1438/2844] 10.222.0.33 () {58 vars in 967 bytes} [Sun Oct 9 01:06:10 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 09 01:06:10.892570 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85952664-7fbc-43c6-bdf7-5d2af17c674d service neutron] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:06:10.894606 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-85952664-7fbc-43c6-bdf7-5d2af17c674d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:06:10.972257 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:06:10.997010 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85952664-7fbc-43c6-bdf7-5d2af17c674d service neutron] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:06:11.002381 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28434fc3-25f5-4f91-b043-199a1783971c service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '70e367e802894b08afd7d756d8d9ecff', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:06:11.002996 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28434fc3-25f5-4f91-b043-199a1783971c service swift] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:06:11.003737 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28434fc3-25f5-4f91-b043-199a1783971c service swift] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:06:11.003737 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28434fc3-25f5-4f91-b043-199a1783971c service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:06:11.016171 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28434fc3-25f5-4f91-b043-199a1783971c service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:06:11.017928 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28434fc3-25f5-4f91-b043-199a1783971c service swift] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:06:11.027514 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1407/2845] 10.222.0.33 () {60 vars in 1384 bytes} [Sun Oct 9 01:06:10 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 135 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 09 01:06:12.203830 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c12386a-0d24-40b4-af9f-e7cc732ff8d8 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:06:12.205240 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-4c12386a-0d24-40b4-af9f-e7cc732ff8d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:06:12.286067 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c12386a-0d24-40b4-af9f-e7cc732ff8d8 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:06:12.291617 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41ca66f7-0fdd-4f19-8a91-2e4943d96a21 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd25e240cb93643ed8122ccac9586936f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:06:12.292301 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ca66f7-0fdd-4f19-8a91-2e4943d96a21 service placement] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:06:12.292583 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ca66f7-0fdd-4f19-8a91-2e4943d96a21 service placement] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:06:12.292855 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ca66f7-0fdd-4f19-8a91-2e4943d96a21 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:06:12.294573 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-41ca66f7-0fdd-4f19-8a91-2e4943d96a21 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:06:12.307239 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41ca66f7-0fdd-4f19-8a91-2e4943d96a21 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:06:12.309121 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41ca66f7-0fdd-4f19-8a91-2e4943d96a21 service placement] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:06:12.319922 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1439/2846] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 01:06:12 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 117 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 01:06:15.444430 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28434fc3-25f5-4f91-b043-199a1783971c service swift] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:06:15.446318 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-28434fc3-25f5-4f91-b043-199a1783971c service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:06:15.471938 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28434fc3-25f5-4f91-b043-199a1783971c service swift] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:06:15.477668 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb097f5a-7bb3-4529-8c8e-afe0a61783e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:06:15.478606 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb097f5a-7bb3-4529-8c8e-afe0a61783e0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:06:15.478963 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb097f5a-7bb3-4529-8c8e-afe0a61783e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:06:15.479310 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb097f5a-7bb3-4529-8c8e-afe0a61783e0 admin admin] PATH_INFO: `/v3/users/04bc22d558c2466ba43a456d08bdd6bd` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:06:15.480577 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb097f5a-7bb3-4529-8c8e-afe0a61783e0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=04bc22d558c2466ba43a456d08bdd6bd)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:06:15.482567 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb097f5a-7bb3-4529-8c8e-afe0a61783e0 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:06:15.529908 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-cb097f5a-7bb3-4529-8c8e-afe0a61783e0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '04bc22d558c2466ba43a456d08bdd6bd'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:06:15.536764 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-cb097f5a-7bb3-4529-8c8e-afe0a61783e0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '04bc22d558c2466ba43a456d08bdd6bd'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:06:15.537424 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-cb097f5a-7bb3-4529-8c8e-afe0a61783e0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '04bc22d558c2466ba43a456d08bdd6bd'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:06:15.544900 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-cb097f5a-7bb3-4529-8c8e-afe0a61783e0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '04bc22d558c2466ba43a456d08bdd6bd'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:06:15.558761 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-cb097f5a-7bb3-4529-8c8e-afe0a61783e0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '04bc22d558c2466ba43a456d08bdd6bd'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:06:15.565229 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1408/2847] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 01:06:15 2022] DELETE /identity/v3/users/04bc22d558c2466ba43a456d08bdd6bd => generated 0 bytes in 121 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:06:15.735988 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41ca66f7-0fdd-4f19-8a91-2e4943d96a21 service placement] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:06:15.747237 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41ca66f7-0fdd-4f19-8a91-2e4943d96a21 service placement] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:06:15.804485 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51003b84-fc16-4b5c-b2bb-56f40069f306 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:06:15.805138 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51003b84-fc16-4b5c-b2bb-56f40069f306 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:06:15.805419 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51003b84-fc16-4b5c-b2bb-56f40069f306 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:06:15.805692 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51003b84-fc16-4b5c-b2bb-56f40069f306 admin admin] PATH_INFO: `/v3/projects/28bc82345ea24b929b66fbcded2b97b7` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:06:15.806798 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51003b84-fc16-4b5c-b2bb-56f40069f306 admin admin] RBAC: Authorizing `identity:delete_project(project_id=28bc82345ea24b929b66fbcded2b97b7)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:06:15.808743 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51003b84-fc16-4b5c-b2bb-56f40069f306 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:06:15.810876 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=100 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:06:15.856485 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-51003b84-fc16-4b5c-b2bb-56f40069f306 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '28bc82345ea24b929b66fbcded2b97b7', 'request_id': 'req-51003b84-fc16-4b5c-b2bb-56f40069f306'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:06:15.861904 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-51003b84-fc16-4b5c-b2bb-56f40069f306 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '28bc82345ea24b929b66fbcded2b97b7', 'request_id': 'req-51003b84-fc16-4b5c-b2bb-56f40069f306'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:06:15.868020 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-51003b84-fc16-4b5c-b2bb-56f40069f306 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '28bc82345ea24b929b66fbcded2b97b7', 'request_id': 'req-51003b84-fc16-4b5c-b2bb-56f40069f306'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:06:15.880456 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-51003b84-fc16-4b5c-b2bb-56f40069f306 admin admin] The token cache is being invalidate because project 28bc82345ea24b929b66fbcded2b97b7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 01:06:15.880740 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-51003b84-fc16-4b5c-b2bb-56f40069f306 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:06:15.881372 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-51003b84-fc16-4b5c-b2bb-56f40069f306 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:06:15.882819 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1440/2848] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 01:06:15 2022] DELETE /identity/v3/projects/28bc82345ea24b929b66fbcded2b97b7 => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:06:15.886978 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb097f5a-7bb3-4529-8c8e-afe0a61783e0 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:06:15.888444 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-cb097f5a-7bb3-4529-8c8e-afe0a61783e0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:06:15.902356 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb097f5a-7bb3-4529-8c8e-afe0a61783e0 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:06:15.960307 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6921d145-6b2b-4da0-97cc-6e57589fc39e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:06:15.960867 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6921d145-6b2b-4da0-97cc-6e57589fc39e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:06:15.961162 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6921d145-6b2b-4da0-97cc-6e57589fc39e admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:06:15.961944 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6921d145-6b2b-4da0-97cc-6e57589fc39e admin admin] PATH_INFO: `/v3/users/88f506c18af44d3e8ea2f64bd627d804` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:06:15.963048 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6921d145-6b2b-4da0-97cc-6e57589fc39e admin admin] RBAC: Authorizing `identity:delete_user(user_id=88f506c18af44d3e8ea2f64bd627d804)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:06:15.964978 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6921d145-6b2b-4da0-97cc-6e57589fc39e admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:06:16.010646 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-6921d145-6b2b-4da0-97cc-6e57589fc39e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '88f506c18af44d3e8ea2f64bd627d804'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:06:16.012639 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=82,keystone:DELETE=17,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:06:16.019285 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-6921d145-6b2b-4da0-97cc-6e57589fc39e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '88f506c18af44d3e8ea2f64bd627d804'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:06:16.020199 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-6921d145-6b2b-4da0-97cc-6e57589fc39e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '88f506c18af44d3e8ea2f64bd627d804'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:06:16.027663 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-6921d145-6b2b-4da0-97cc-6e57589fc39e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '88f506c18af44d3e8ea2f64bd627d804'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:06:16.039750 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-6921d145-6b2b-4da0-97cc-6e57589fc39e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '88f506c18af44d3e8ea2f64bd627d804'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:06:16.045696 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1409/2849] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 01:06:15 2022] DELETE /identity/v3/users/88f506c18af44d3e8ea2f64bd627d804 => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:06:16.215467 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51003b84-fc16-4b5c-b2bb-56f40069f306 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:06:16.216983 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-51003b84-fc16-4b5c-b2bb-56f40069f306 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:06:16.229953 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51003b84-fc16-4b5c-b2bb-56f40069f306 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:06:16.286117 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aae6f12-53b7-4999-89f4-bf5dada7ffc7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:06:16.286729 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aae6f12-53b7-4999-89f4-bf5dada7ffc7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:06:16.287011 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aae6f12-53b7-4999-89f4-bf5dada7ffc7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:06:16.287282 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aae6f12-53b7-4999-89f4-bf5dada7ffc7 admin admin] PATH_INFO: `/v3/projects/cac43224e7954c44a71839707ffcde6c` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:06:16.288409 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aae6f12-53b7-4999-89f4-bf5dada7ffc7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=cac43224e7954c44a71839707ffcde6c)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:06:16.290366 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aae6f12-53b7-4999-89f4-bf5dada7ffc7 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:06:16.335097 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-4aae6f12-53b7-4999-89f4-bf5dada7ffc7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cac43224e7954c44a71839707ffcde6c', 'request_id': 'req-4aae6f12-53b7-4999-89f4-bf5dada7ffc7'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:06:16.341100 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-4aae6f12-53b7-4999-89f4-bf5dada7ffc7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cac43224e7954c44a71839707ffcde6c', 'request_id': 'req-4aae6f12-53b7-4999-89f4-bf5dada7ffc7'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:06:16.347147 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-4aae6f12-53b7-4999-89f4-bf5dada7ffc7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cac43224e7954c44a71839707ffcde6c', 'request_id': 'req-4aae6f12-53b7-4999-89f4-bf5dada7ffc7'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:06:16.358806 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-4aae6f12-53b7-4999-89f4-bf5dada7ffc7 admin admin] The token cache is being invalidate because project cac43224e7954c44a71839707ffcde6c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 01:06:16.359096 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-4aae6f12-53b7-4999-89f4-bf5dada7ffc7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:06:16.359753 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-4aae6f12-53b7-4999-89f4-bf5dada7ffc7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:06:16.361246 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1441/2850] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 01:06:16 2022] DELETE /identity/v3/projects/cac43224e7954c44a71839707ffcde6c => generated 0 bytes in 146 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 09 01:06:26.042302 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=11,keystone:INSERT=1,keystone:DELETE=3 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:06:26.355859 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=74,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:06:43.741875 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6921d145-6b2b-4da0-97cc-6e57589fc39e admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:06:43.743323 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-6921d145-6b2b-4da0-97cc-6e57589fc39e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:06:43.819167 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6921d145-6b2b-4da0-97cc-6e57589fc39e admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:06:43.865434 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-460f8d4a-0b46-4909-9b13-17a2bb8cd155 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3f48335668a4e8f934ac11ff66cd308', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:06:43.866025 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-460f8d4a-0b46-4909-9b13-17a2bb8cd155 service cinder] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:06:43.866301 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-460f8d4a-0b46-4909-9b13-17a2bb8cd155 service cinder] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:06:43.866570 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-460f8d4a-0b46-4909-9b13-17a2bb8cd155 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:06:43.868312 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-460f8d4a-0b46-4909-9b13-17a2bb8cd155 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:06:43.925690 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-460f8d4a-0b46-4909-9b13-17a2bb8cd155 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:06:43.927114 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-460f8d4a-0b46-4909-9b13-17a2bb8cd155 service cinder] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:06:43.935949 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1410/2851] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 01:06:43 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 195 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 01:06:46.595255 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aae6f12-53b7-4999-89f4-bf5dada7ffc7 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:06:46.597344 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-4aae6f12-53b7-4999-89f4-bf5dada7ffc7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:06:46.710514 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aae6f12-53b7-4999-89f4-bf5dada7ffc7 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:06:46.717256 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80b5ab06-6735-435d-9361-30d2d09c5fd7 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a5d95969e74157bb2d2a57207e1651', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:06:46.718079 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80b5ab06-6735-435d-9361-30d2d09c5fd7 service nova] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:06:46.718515 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80b5ab06-6735-435d-9361-30d2d09c5fd7 service nova] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:06:46.718822 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80b5ab06-6735-435d-9361-30d2d09c5fd7 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:06:46.721261 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-80b5ab06-6735-435d-9361-30d2d09c5fd7 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:06:46.736656 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80b5ab06-6735-435d-9361-30d2d09c5fd7 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:06:46.738826 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80b5ab06-6735-435d-9361-30d2d09c5fd7 service nova] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:06:46.755204 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1442/2852] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 01:06:46 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 160 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 01:06:50.764079 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-460f8d4a-0b46-4909-9b13-17a2bb8cd155 service cinder] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:06:50.765531 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-460f8d4a-0b46-4909-9b13-17a2bb8cd155 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:06:50.820864 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-460f8d4a-0b46-4909-9b13-17a2bb8cd155 service cinder] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:06:50.824553 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc6dbcde-43cb-480b-8606-3b6d1a87b2ca service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:06:50.825152 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc6dbcde-43cb-480b-8606-3b6d1a87b2ca service neutron] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:06:50.825427 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc6dbcde-43cb-480b-8606-3b6d1a87b2ca service neutron] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:06:50.825694 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc6dbcde-43cb-480b-8606-3b6d1a87b2ca service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:06:50.827411 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-fc6dbcde-43cb-480b-8606-3b6d1a87b2ca service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:06:50.832141 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=100 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:06:50.889540 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc6dbcde-43cb-480b-8606-3b6d1a87b2ca service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:06:50.890951 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc6dbcde-43cb-480b-8606-3b6d1a87b2ca service neutron] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:06:50.901928 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1411/2853] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 01:06:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 138 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 01:06:53.990207 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80b5ab06-6735-435d-9361-30d2d09c5fd7 service nova] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:06:53.991736 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-80b5ab06-6735-435d-9361-30d2d09c5fd7 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:06:54.006204 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80b5ab06-6735-435d-9361-30d2d09c5fd7 service nova] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:06:54.010831 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a41bb4c-cff5-450b-87a6-cf77aa7ef0c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:06:54.011448 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a41bb4c-cff5-450b-87a6-cf77aa7ef0c9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:06:54.011789 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a41bb4c-cff5-450b-87a6-cf77aa7ef0c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:06:54.012091 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a41bb4c-cff5-450b-87a6-cf77aa7ef0c9 admin admin] PATH_INFO: `/v3/users/56b5d764f0044358ac594a2edd6f4c0f` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:06:54.013198 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a41bb4c-cff5-450b-87a6-cf77aa7ef0c9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=56b5d764f0044358ac594a2edd6f4c0f)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:06:54.015157 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a41bb4c-cff5-450b-87a6-cf77aa7ef0c9 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:06:54.061912 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-0a41bb4c-cff5-450b-87a6-cf77aa7ef0c9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '56b5d764f0044358ac594a2edd6f4c0f'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:06:54.068914 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-0a41bb4c-cff5-450b-87a6-cf77aa7ef0c9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '56b5d764f0044358ac594a2edd6f4c0f'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:06:54.069592 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-0a41bb4c-cff5-450b-87a6-cf77aa7ef0c9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '56b5d764f0044358ac594a2edd6f4c0f'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:06:54.070343 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-0a41bb4c-cff5-450b-87a6-cf77aa7ef0c9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '56b5d764f0044358ac594a2edd6f4c0f'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:06:54.078121 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-0a41bb4c-cff5-450b-87a6-cf77aa7ef0c9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '56b5d764f0044358ac594a2edd6f4c0f'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:06:54.096929 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1443/2854] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 01:06:53 2022] DELETE /identity/v3/users/56b5d764f0044358ac594a2edd6f4c0f => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:06:54.268120 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc6dbcde-43cb-480b-8606-3b6d1a87b2ca service neutron] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:06:54.278521 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc6dbcde-43cb-480b-8606-3b6d1a87b2ca service neutron] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:06:54.335444 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67462adc-b35e-4aa6-8b5c-22ef84c4fd8b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:06:54.336658 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67462adc-b35e-4aa6-8b5c-22ef84c4fd8b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:06:54.336940 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67462adc-b35e-4aa6-8b5c-22ef84c4fd8b admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:06:54.337214 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67462adc-b35e-4aa6-8b5c-22ef84c4fd8b admin admin] PATH_INFO: `/v3/projects/410212da4a794c07b30ed22cc5b7e760` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:06:54.338322 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67462adc-b35e-4aa6-8b5c-22ef84c4fd8b admin admin] RBAC: Authorizing `identity:delete_project(project_id=410212da4a794c07b30ed22cc5b7e760)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:06:54.340251 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67462adc-b35e-4aa6-8b5c-22ef84c4fd8b admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:06:54.386732 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-67462adc-b35e-4aa6-8b5c-22ef84c4fd8b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '410212da4a794c07b30ed22cc5b7e760', 'request_id': 'req-67462adc-b35e-4aa6-8b5c-22ef84c4fd8b'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:06:54.392317 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-67462adc-b35e-4aa6-8b5c-22ef84c4fd8b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '410212da4a794c07b30ed22cc5b7e760', 'request_id': 'req-67462adc-b35e-4aa6-8b5c-22ef84c4fd8b'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:06:54.399002 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-67462adc-b35e-4aa6-8b5c-22ef84c4fd8b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '410212da4a794c07b30ed22cc5b7e760', 'request_id': 'req-67462adc-b35e-4aa6-8b5c-22ef84c4fd8b'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:06:54.411265 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-67462adc-b35e-4aa6-8b5c-22ef84c4fd8b admin admin] The token cache is being invalidate because project 410212da4a794c07b30ed22cc5b7e760 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81453) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 01:06:54.411552 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-67462adc-b35e-4aa6-8b5c-22ef84c4fd8b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:06:54.412180 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-67462adc-b35e-4aa6-8b5c-22ef84c4fd8b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:06:54.413675 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1412/2855] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 01:06:54 2022] DELETE /identity/v3/projects/410212da4a794c07b30ed22cc5b7e760 => generated 0 bytes in 146 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:06:54.418777 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4117952d-18f7-4556-bc30-270c04978982 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:06:54.419087 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4117952d-18f7-4556-bc30-270c04978982 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:06:54.419363 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4117952d-18f7-4556-bc30-270c04978982 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:06:54.475732 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-4117952d-18f7-4556-bc30-270c04978982 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:06:54.477359 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-4117952d-18f7-4556-bc30-270c04978982 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:06:54.497934 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:06:54.538324 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1444/2856] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:06:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:06:54.546700 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d3ea157-534c-4fb0-a617-911784a35593 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:06:54.546985 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d3ea157-534c-4fb0-a617-911784a35593 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:06:54.547262 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d3ea157-534c-4fb0-a617-911784a35593 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:06:54.567314 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-3d3ea157-534c-4fb0-a617-911784a35593 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:06:54.568562 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-3d3ea157-534c-4fb0-a617-911784a35593 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:06:54.576802 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1413/2857] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:06:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:06:54.581002 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4117952d-18f7-4556-bc30-270c04978982 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:06:54.591018 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4117952d-18f7-4556-bc30-270c04978982 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:06:54.595479 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58dccfa4-76fb-41f9-b3ca-cf912e5d1ac0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:06:54.596100 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58dccfa4-76fb-41f9-b3ca-cf912e5d1ac0 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:06:54.596376 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58dccfa4-76fb-41f9-b3ca-cf912e5d1ac0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:06:54.596643 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58dccfa4-76fb-41f9-b3ca-cf912e5d1ac0 admin admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:06:54.597272 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58dccfa4-76fb-41f9-b3ca-cf912e5d1ac0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 01:06:54.597567 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58dccfa4-76fb-41f9-b3ca-cf912e5d1ac0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:06:54.599127 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58dccfa4-76fb-41f9-b3ca-cf912e5d1ac0 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:06:54.610731 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1445/2858] 10.222.0.33 () {58 vars in 1153 bytes} [Sun Oct 9 01:06:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 01:06:54.614805 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d3ea157-534c-4fb0-a617-911784a35593 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:06:54.624332 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d3ea157-534c-4fb0-a617-911784a35593 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:06:54.628981 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0171f514-3617-4fba-87a7-fad9c35e31e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:06:54.629559 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0171f514-3617-4fba-87a7-fad9c35e31e4 admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:06:54.629839 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0171f514-3617-4fba-87a7-fad9c35e31e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:06:54.630108 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0171f514-3617-4fba-87a7-fad9c35e31e4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:06:54.630732 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0171f514-3617-4fba-87a7-fad9c35e31e4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:06:54.633179 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0171f514-3617-4fba-87a7-fad9c35e31e4 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:06:54.656570 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:06:54.669580 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1414/2859] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 01:06:54 2022] POST /identity/v3/projects => generated 373 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 01:06:54.673738 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58dccfa4-76fb-41f9-b3ca-cf912e5d1ac0 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:06:54.684606 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58dccfa4-76fb-41f9-b3ca-cf912e5d1ac0 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:06:54.738893 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6116b2d-0006-4df3-93bb-3104cbe195c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:06:54.739409 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6116b2d-0006-4df3-93bb-3104cbe195c1 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:06:54.739715 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6116b2d-0006-4df3-93bb-3104cbe195c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:06:54.739987 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6116b2d-0006-4df3-93bb-3104cbe195c1 admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:06:54.740597 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6116b2d-0006-4df3-93bb-3104cbe195c1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:06:54.744554 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6116b2d-0006-4df3-93bb-3104cbe195c1 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:06:54.765544 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1446/2860] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 01:06:54 2022] POST /identity/v3/users => generated 287 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 09 01:06:54.769610 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0171f514-3617-4fba-87a7-fad9c35e31e4 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:06:54.779894 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0171f514-3617-4fba-87a7-fad9c35e31e4 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:06:54.784567 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a309e20e-6831-48e0-94ff-381de934d6b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:06:54.785221 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a309e20e-6831-48e0-94ff-381de934d6b5 admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:06:54.785501 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a309e20e-6831-48e0-94ff-381de934d6b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:06:54.785771 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a309e20e-6831-48e0-94ff-381de934d6b5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:06:54.786395 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a309e20e-6831-48e0-94ff-381de934d6b5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:06:54.788017 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a309e20e-6831-48e0-94ff-381de934d6b5 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:06:54.797618 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1415/2861] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 01:06:54 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 01:06:54.801355 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6116b2d-0006-4df3-93bb-3104cbe195c1 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:06:54.811772 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6116b2d-0006-4df3-93bb-3104cbe195c1 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:06:54.816445 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-581c40b4-d1fa-406a-bcf3-def4401de9ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:06:54.817032 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-581c40b4-d1fa-406a-bcf3-def4401de9ed admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:06:54.817317 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-581c40b4-d1fa-406a-bcf3-def4401de9ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:06:54.817595 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-581c40b4-d1fa-406a-bcf3-def4401de9ed admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:06:54.818230 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-581c40b4-d1fa-406a-bcf3-def4401de9ed admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:06:54.819856 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-581c40b4-d1fa-406a-bcf3-def4401de9ed admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:06:54.830913 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1447/2862] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 01:06:54 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 01:06:54.834971 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a309e20e-6831-48e0-94ff-381de934d6b5 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:06:54.844793 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a309e20e-6831-48e0-94ff-381de934d6b5 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:06:54.849549 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fee78eda-dd8b-40a3-924d-cf81846eddb0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:06:54.850196 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee78eda-dd8b-40a3-924d-cf81846eddb0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:06:54.850477 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee78eda-dd8b-40a3-924d-cf81846eddb0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:06:54.850747 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee78eda-dd8b-40a3-924d-cf81846eddb0 admin admin] PATH_INFO: `/v3/projects/26f33282f4974f2580c0174fa49ba422/users/0daf7cc1dd5846cdb8895e543fc031b7/roles/588016933bab436d8b7c3ac76827356d` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:06:54.865169 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fee78eda-dd8b-40a3-924d-cf81846eddb0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=26f33282f4974f2580c0174fa49ba422, user_id=0daf7cc1dd5846cdb8895e543fc031b7, role_id=588016933bab436d8b7c3ac76827356d)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:06:54.868336 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fee78eda-dd8b-40a3-924d-cf81846eddb0 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:06:54.880667 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1416/2863] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 01:06:54 2022] PUT /identity/v3/projects/26f33282f4974f2580c0174fa49ba422/users/0daf7cc1dd5846cdb8895e543fc031b7/roles/588016933bab436d8b7c3ac76827356d => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:06:54.892367 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c489df8-78ef-4119-8e51-1325ae20c106 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:06:54.892657 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c489df8-78ef-4119-8e51-1325ae20c106 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:06:54.892931 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c489df8-78ef-4119-8e51-1325ae20c106 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:06:54.921177 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-1c489df8-78ef-4119-8e51-1325ae20c106 None None] MFA Rules not processed for user `0daf7cc1dd5846cdb8895e543fc031b7`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:06:54.922365 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-1c489df8-78ef-4119-8e51-1325ae20c106 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:06:54.987106 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1448/2864] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:06:54 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:06:54.991495 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fee78eda-dd8b-40a3-924d-cf81846eddb0 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:06:55.052419 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fee78eda-dd8b-40a3-924d-cf81846eddb0 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:06:55.056968 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8707b168-5647-4245-ba0d-98bae406edfa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:06:55.057550 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8707b168-5647-4245-ba0d-98bae406edfa admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:06:55.057826 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8707b168-5647-4245-ba0d-98bae406edfa admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:06:55.058095 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8707b168-5647-4245-ba0d-98bae406edfa admin admin] PATH_INFO: `/v3/projects` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:06:55.058710 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8707b168-5647-4245-ba0d-98bae406edfa admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:06:55.060626 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8707b168-5647-4245-ba0d-98bae406edfa admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:06:55.088024 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1417/2865] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 01:06:54 2022] POST /identity/v3/projects => generated 373 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 01:06:55.092050 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c489df8-78ef-4119-8e51-1325ae20c106 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:06:55.102740 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c489df8-78ef-4119-8e51-1325ae20c106 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:06:55.120467 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:06:55.161545 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8ee803e-2c52-47b4-a0fc-7a4396d66e33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:06:55.162139 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ee803e-2c52-47b4-a0fc-7a4396d66e33 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:06:55.162417 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ee803e-2c52-47b4-a0fc-7a4396d66e33 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:06:55.162687 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ee803e-2c52-47b4-a0fc-7a4396d66e33 admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:06:55.163313 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8ee803e-2c52-47b4-a0fc-7a4396d66e33 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:06:55.165271 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8ee803e-2c52-47b4-a0fc-7a4396d66e33 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:06:55.186257 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1449/2866] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 01:06:55 2022] POST /identity/v3/users => generated 293 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 01:06:55.190164 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8707b168-5647-4245-ba0d-98bae406edfa admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:06:55.200817 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8707b168-5647-4245-ba0d-98bae406edfa admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:06:55.205426 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b47eae-15ef-4b97-86f6-68048795d88a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:06:55.206041 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b47eae-15ef-4b97-86f6-68048795d88a admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:06:55.206319 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b47eae-15ef-4b97-86f6-68048795d88a admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:06:55.206590 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b47eae-15ef-4b97-86f6-68048795d88a admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:06:55.207218 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07b47eae-15ef-4b97-86f6-68048795d88a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:06:55.208832 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07b47eae-15ef-4b97-86f6-68048795d88a admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:06:55.219526 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1418/2867] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 01:06:55 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 09 01:06:55.223616 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8ee803e-2c52-47b4-a0fc-7a4396d66e33 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:06:55.233259 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8ee803e-2c52-47b4-a0fc-7a4396d66e33 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:06:55.237749 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79d33c64-9230-4036-a651-ed3990e45fea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:06:55.238368 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d33c64-9230-4036-a651-ed3990e45fea admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:06:55.238642 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d33c64-9230-4036-a651-ed3990e45fea admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:06:55.238908 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d33c64-9230-4036-a651-ed3990e45fea admin admin] PATH_INFO: `/v3/projects/d6b4b6ece1084307b8c102eeaacd1812/users/1d40abc3916844b2958631d7a37e3073/roles/092036ee46ac427baf4d56468575f697` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:06:55.253341 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79d33c64-9230-4036-a651-ed3990e45fea admin admin] RBAC: Authorizing `identity:create_grant(project_id=d6b4b6ece1084307b8c102eeaacd1812, user_id=1d40abc3916844b2958631d7a37e3073, role_id=092036ee46ac427baf4d56468575f697)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:06:55.255915 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79d33c64-9230-4036-a651-ed3990e45fea admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:06:55.268669 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1450/2868] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 01:06:55 2022] PUT /identity/v3/projects/d6b4b6ece1084307b8c102eeaacd1812/users/1d40abc3916844b2958631d7a37e3073/roles/092036ee46ac427baf4d56468575f697 => generated 0 bytes in 45 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:06:55.272612 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b47eae-15ef-4b97-86f6-68048795d88a admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:06:55.283318 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b47eae-15ef-4b97-86f6-68048795d88a admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:06:55.339560 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16817061-5122-4402-9f8e-0bb8b212370c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:06:55.340196 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16817061-5122-4402-9f8e-0bb8b212370c admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:06:55.340474 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16817061-5122-4402-9f8e-0bb8b212370c admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:06:55.340862 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16817061-5122-4402-9f8e-0bb8b212370c admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:06:55.341497 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16817061-5122-4402-9f8e-0bb8b212370c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:06:55.343107 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16817061-5122-4402-9f8e-0bb8b212370c admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:06:55.353183 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1419/2869] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 01:06:55 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 01:06:55.357151 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79d33c64-9230-4036-a651-ed3990e45fea admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:06:55.367537 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79d33c64-9230-4036-a651-ed3990e45fea admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:06:55.372042 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81159ff7-69da-46e8-923b-cf144f201d6e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:06:55.372675 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81159ff7-69da-46e8-923b-cf144f201d6e admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:06:55.372948 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81159ff7-69da-46e8-923b-cf144f201d6e admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:06:55.373217 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81159ff7-69da-46e8-923b-cf144f201d6e admin admin] PATH_INFO: `/v3/projects/d6b4b6ece1084307b8c102eeaacd1812/users/1d40abc3916844b2958631d7a37e3073/roles/092036ee46ac427baf4d56468575f697` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:06:55.375136 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81159ff7-69da-46e8-923b-cf144f201d6e admin admin] RBAC: Authorizing `identity:create_grant(project_id=d6b4b6ece1084307b8c102eeaacd1812, user_id=1d40abc3916844b2958631d7a37e3073, role_id=092036ee46ac427baf4d56468575f697)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:06:55.377785 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81159ff7-69da-46e8-923b-cf144f201d6e admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:06:55.388680 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1451/2870] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 01:06:55 2022] PUT /identity/v3/projects/d6b4b6ece1084307b8c102eeaacd1812/users/1d40abc3916844b2958631d7a37e3073/roles/092036ee46ac427baf4d56468575f697 => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:06:55.399061 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-951d04da-af4f-4339-a6d8-90aff25ded40 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:06:55.399343 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-951d04da-af4f-4339-a6d8-90aff25ded40 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:06:55.399641 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-951d04da-af4f-4339-a6d8-90aff25ded40 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:06:55.426883 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-951d04da-af4f-4339-a6d8-90aff25ded40 None None] MFA Rules not processed for user `1d40abc3916844b2958631d7a37e3073`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:06:55.428109 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-951d04da-af4f-4339-a6d8-90aff25ded40 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:06:55.443838 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:INSERT=3,keystone:SELECT=97 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:06:55.494962 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1420/2871] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:06:55 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:06:55.506016 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81159ff7-69da-46e8-923b-cf144f201d6e admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:06:55.507817 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-81159ff7-69da-46e8-923b-cf144f201d6e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:06:55.565318 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81159ff7-69da-46e8-923b-cf144f201d6e admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:06:55.569261 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-148dd7a8-56cd-4834-89f0-4e449f864df1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3f48335668a4e8f934ac11ff66cd308', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:06:55.569854 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148dd7a8-56cd-4834-89f0-4e449f864df1 service cinder] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:06:55.570205 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148dd7a8-56cd-4834-89f0-4e449f864df1 service cinder] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:06:55.570473 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148dd7a8-56cd-4834-89f0-4e449f864df1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:06:55.579369 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-148dd7a8-56cd-4834-89f0-4e449f864df1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:06:55.580921 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-148dd7a8-56cd-4834-89f0-4e449f864df1 service cinder] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:06:55.590922 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1452/2872] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 01:06:55 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 01:06:55.968115 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-951d04da-af4f-4339-a6d8-90aff25ded40 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:06:56.029965 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-951d04da-af4f-4339-a6d8-90aff25ded40 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:06:56.034371 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-357b6f41-54b8-4857-89fd-788f0feaa372 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:06:56.034989 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-357b6f41-54b8-4857-89fd-788f0feaa372 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:06:56.035265 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-357b6f41-54b8-4857-89fd-788f0feaa372 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:06:56.035532 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-357b6f41-54b8-4857-89fd-788f0feaa372 admin admin] PATH_INFO: `/v3/users/0daf7cc1dd5846cdb8895e543fc031b7` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:06:56.036720 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-357b6f41-54b8-4857-89fd-788f0feaa372 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0daf7cc1dd5846cdb8895e543fc031b7)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:06:56.038621 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-357b6f41-54b8-4857-89fd-788f0feaa372 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:06:56.083741 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-357b6f41-54b8-4857-89fd-788f0feaa372 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0daf7cc1dd5846cdb8895e543fc031b7'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:06:56.091478 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-357b6f41-54b8-4857-89fd-788f0feaa372 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0daf7cc1dd5846cdb8895e543fc031b7'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:06:56.092208 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-357b6f41-54b8-4857-89fd-788f0feaa372 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0daf7cc1dd5846cdb8895e543fc031b7'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:06:56.099749 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-357b6f41-54b8-4857-89fd-788f0feaa372 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0daf7cc1dd5846cdb8895e543fc031b7'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:06:56.111539 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-357b6f41-54b8-4857-89fd-788f0feaa372 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0daf7cc1dd5846cdb8895e543fc031b7'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:06:56.117925 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1421/2873] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 01:06:55 2022] DELETE /identity/v3/users/0daf7cc1dd5846cdb8895e543fc031b7 => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:06:56.122348 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de81556-bbd3-4fba-b15a-498b58c5d07c None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:06:56.122826 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de81556-bbd3-4fba-b15a-498b58c5d07c None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:06:56.122946 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de81556-bbd3-4fba-b15a-498b58c5d07c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:06:56.141893 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-9de81556-bbd3-4fba-b15a-498b58c5d07c None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:06:56.143085 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-9de81556-bbd3-4fba-b15a-498b58c5d07c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:06:56.200413 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1453/2874] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:06:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:06:56.210205 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-357b6f41-54b8-4857-89fd-788f0feaa372 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:06:56.211657 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-357b6f41-54b8-4857-89fd-788f0feaa372 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:06:56.246038 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:06:56.268896 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-357b6f41-54b8-4857-89fd-788f0feaa372 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:06:56.272551 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19c9d903-ba62-41b6-9ae8-401b71a44e23 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:06:56.273175 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c9d903-ba62-41b6-9ae8-401b71a44e23 service neutron] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:06:56.273726 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c9d903-ba62-41b6-9ae8-401b71a44e23 service neutron] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:06:56.273726 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c9d903-ba62-41b6-9ae8-401b71a44e23 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:06:56.282765 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19c9d903-ba62-41b6-9ae8-401b71a44e23 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:06:56.284168 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19c9d903-ba62-41b6-9ae8-401b71a44e23 service neutron] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:06:56.293489 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1422/2875] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 01:06:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 01:06:56.556559 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9de81556-bbd3-4fba-b15a-498b58c5d07c None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:06:56.558001 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-9de81556-bbd3-4fba-b15a-498b58c5d07c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:06:56.571871 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9de81556-bbd3-4fba-b15a-498b58c5d07c None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:06:56.576559 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03551bb9-f97d-4f9e-9e6a-2c1c92df90b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:06:56.577182 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03551bb9-f97d-4f9e-9e6a-2c1c92df90b7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:06:56.577457 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03551bb9-f97d-4f9e-9e6a-2c1c92df90b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:06:56.577724 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03551bb9-f97d-4f9e-9e6a-2c1c92df90b7 admin admin] PATH_INFO: `/v3/projects/26f33282f4974f2580c0174fa49ba422` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:06:56.578821 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03551bb9-f97d-4f9e-9e6a-2c1c92df90b7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=26f33282f4974f2580c0174fa49ba422)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:06:56.580767 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03551bb9-f97d-4f9e-9e6a-2c1c92df90b7 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:06:56.593702 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:06:56.628540 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-03551bb9-f97d-4f9e-9e6a-2c1c92df90b7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '26f33282f4974f2580c0174fa49ba422', 'request_id': 'req-03551bb9-f97d-4f9e-9e6a-2c1c92df90b7'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:06:56.634466 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-03551bb9-f97d-4f9e-9e6a-2c1c92df90b7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '26f33282f4974f2580c0174fa49ba422', 'request_id': 'req-03551bb9-f97d-4f9e-9e6a-2c1c92df90b7'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:06:56.640528 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-03551bb9-f97d-4f9e-9e6a-2c1c92df90b7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '26f33282f4974f2580c0174fa49ba422', 'request_id': 'req-03551bb9-f97d-4f9e-9e6a-2c1c92df90b7'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:06:56.652290 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-03551bb9-f97d-4f9e-9e6a-2c1c92df90b7 admin admin] The token cache is being invalidate because project 26f33282f4974f2580c0174fa49ba422 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 01:06:56.652573 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-03551bb9-f97d-4f9e-9e6a-2c1c92df90b7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:06:56.653178 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-03551bb9-f97d-4f9e-9e6a-2c1c92df90b7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:06:56.655357 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1454/2876] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 01:06:56 2022] DELETE /identity/v3/projects/26f33282f4974f2580c0174fa49ba422 => generated 0 bytes in 99 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:06:56.659304 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19c9d903-ba62-41b6-9ae8-401b71a44e23 service neutron] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:06:56.668598 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19c9d903-ba62-41b6-9ae8-401b71a44e23 service neutron] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:06:56.723097 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aed8bd7a-9acd-4157-a6b5-cdca7f50a020 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:06:56.723751 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed8bd7a-9acd-4157-a6b5-cdca7f50a020 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:06:56.724034 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed8bd7a-9acd-4157-a6b5-cdca7f50a020 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:06:56.724306 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed8bd7a-9acd-4157-a6b5-cdca7f50a020 admin admin] PATH_INFO: `/v3/users/1d40abc3916844b2958631d7a37e3073` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:06:56.725403 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aed8bd7a-9acd-4157-a6b5-cdca7f50a020 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1d40abc3916844b2958631d7a37e3073)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:06:56.727294 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aed8bd7a-9acd-4157-a6b5-cdca7f50a020 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:06:56.772076 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-aed8bd7a-9acd-4157-a6b5-cdca7f50a020 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1d40abc3916844b2958631d7a37e3073'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:06:56.779346 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-aed8bd7a-9acd-4157-a6b5-cdca7f50a020 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1d40abc3916844b2958631d7a37e3073'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:06:56.780021 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-aed8bd7a-9acd-4157-a6b5-cdca7f50a020 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1d40abc3916844b2958631d7a37e3073'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:06:56.787262 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-aed8bd7a-9acd-4157-a6b5-cdca7f50a020 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1d40abc3916844b2958631d7a37e3073'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:06:56.799543 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-aed8bd7a-9acd-4157-a6b5-cdca7f50a020 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1d40abc3916844b2958631d7a37e3073'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:06:56.805797 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1423/2877] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 01:06:56 2022] DELETE /identity/v3/users/1d40abc3916844b2958631d7a37e3073 => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:06:57.072567 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03551bb9-f97d-4f9e-9e6a-2c1c92df90b7 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:06:57.074253 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-03551bb9-f97d-4f9e-9e6a-2c1c92df90b7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:06:57.091324 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03551bb9-f97d-4f9e-9e6a-2c1c92df90b7 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:06:57.147028 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aebeb24b-289e-4acd-ac12-b5b28a7da24c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:06:57.147681 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aebeb24b-289e-4acd-ac12-b5b28a7da24c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:06:57.147947 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aebeb24b-289e-4acd-ac12-b5b28a7da24c admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:06:57.148780 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aebeb24b-289e-4acd-ac12-b5b28a7da24c admin admin] PATH_INFO: `/v3/projects/d6b4b6ece1084307b8c102eeaacd1812` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:06:57.149907 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aebeb24b-289e-4acd-ac12-b5b28a7da24c admin admin] RBAC: Authorizing `identity:delete_project(project_id=d6b4b6ece1084307b8c102eeaacd1812)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:06:57.151860 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aebeb24b-289e-4acd-ac12-b5b28a7da24c admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:06:57.203156 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-aebeb24b-289e-4acd-ac12-b5b28a7da24c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd6b4b6ece1084307b8c102eeaacd1812', 'request_id': 'req-aebeb24b-289e-4acd-ac12-b5b28a7da24c'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:06:57.208723 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-aebeb24b-289e-4acd-ac12-b5b28a7da24c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd6b4b6ece1084307b8c102eeaacd1812', 'request_id': 'req-aebeb24b-289e-4acd-ac12-b5b28a7da24c'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:06:57.215418 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-aebeb24b-289e-4acd-ac12-b5b28a7da24c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd6b4b6ece1084307b8c102eeaacd1812', 'request_id': 'req-aebeb24b-289e-4acd-ac12-b5b28a7da24c'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:06:57.227082 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-aebeb24b-289e-4acd-ac12-b5b28a7da24c admin admin] The token cache is being invalidate because project d6b4b6ece1084307b8c102eeaacd1812 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 01:06:57.227372 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-aebeb24b-289e-4acd-ac12-b5b28a7da24c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:06:57.228013 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-aebeb24b-289e-4acd-ac12-b5b28a7da24c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:06:57.229478 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1455/2878] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 01:06:57 2022] DELETE /identity/v3/projects/d6b4b6ece1084307b8c102eeaacd1812 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:06:57.237334 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4975b160-0d98-4fce-b3b9-062aafaba708 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:06:57.237641 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4975b160-0d98-4fce-b3b9-062aafaba708 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:06:57.237920 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4975b160-0d98-4fce-b3b9-062aafaba708 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:06:57.256985 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-4975b160-0d98-4fce-b3b9-062aafaba708 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:06:57.258744 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-4975b160-0d98-4fce-b3b9-062aafaba708 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:06:57.304690 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:06:57.317463 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1424/2879] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:06:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:06:57.325764 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70370b21-c79c-40ec-9baa-44d7dc4ae88a None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:06:57.326048 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70370b21-c79c-40ec-9baa-44d7dc4ae88a None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:06:57.326322 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70370b21-c79c-40ec-9baa-44d7dc4ae88a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:06:57.346118 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-70370b21-c79c-40ec-9baa-44d7dc4ae88a None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:06:57.347308 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-70370b21-c79c-40ec-9baa-44d7dc4ae88a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:06:57.355104 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1456/2880] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:06:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:06:57.359407 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4975b160-0d98-4fce-b3b9-062aafaba708 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:06:57.369105 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4975b160-0d98-4fce-b3b9-062aafaba708 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:06:57.373611 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a0afcfe-aa70-4e9f-b103-bb971a90a244 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:06:57.374233 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0afcfe-aa70-4e9f-b103-bb971a90a244 admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:06:57.374510 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0afcfe-aa70-4e9f-b103-bb971a90a244 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:06:57.374778 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0afcfe-aa70-4e9f-b103-bb971a90a244 admin admin] PATH_INFO: `/v3/domains` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:06:57.375410 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a0afcfe-aa70-4e9f-b103-bb971a90a244 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81453) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 01:06:57.375747 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a0afcfe-aa70-4e9f-b103-bb971a90a244 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:06:57.377313 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a0afcfe-aa70-4e9f-b103-bb971a90a244 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:06:57.389068 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1425/2881] 10.222.0.33 () {58 vars in 1153 bytes} [Sun Oct 9 01:06:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 01:06:57.393156 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70370b21-c79c-40ec-9baa-44d7dc4ae88a None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:06:57.402786 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70370b21-c79c-40ec-9baa-44d7dc4ae88a None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:06:57.407651 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35930755-3457-40bb-a312-5d4ca3027937 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:06:57.408231 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35930755-3457-40bb-a312-5d4ca3027937 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:06:57.408508 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35930755-3457-40bb-a312-5d4ca3027937 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:06:57.408774 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35930755-3457-40bb-a312-5d4ca3027937 admin admin] PATH_INFO: `/v3/projects` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:06:57.409392 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35930755-3457-40bb-a312-5d4ca3027937 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:06:57.411275 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35930755-3457-40bb-a312-5d4ca3027937 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:06:57.437151 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1457/2882] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 01:06:57 2022] POST /identity/v3/projects => generated 381 bytes in 44 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 01:06:57.441255 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a0afcfe-aa70-4e9f-b103-bb971a90a244 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:06:57.451750 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a0afcfe-aa70-4e9f-b103-bb971a90a244 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:06:57.506959 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec277914-8655-4ff9-ac75-7f51b8048b23 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:06:57.508114 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec277914-8655-4ff9-ac75-7f51b8048b23 admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:06:57.508388 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec277914-8655-4ff9-ac75-7f51b8048b23 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:06:57.508656 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec277914-8655-4ff9-ac75-7f51b8048b23 admin admin] PATH_INFO: `/v3/users` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:06:57.509280 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec277914-8655-4ff9-ac75-7f51b8048b23 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:06:57.511170 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec277914-8655-4ff9-ac75-7f51b8048b23 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:06:57.531935 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1426/2883] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 01:06:57 2022] POST /identity/v3/users => generated 291 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 09 01:06:57.536019 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35930755-3457-40bb-a312-5d4ca3027937 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:06:57.541409 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:06:57.547754 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35930755-3457-40bb-a312-5d4ca3027937 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:06:57.553354 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adc56558-82b9-42b2-b79e-a132fc430a04 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:06:57.553946 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc56558-82b9-42b2-b79e-a132fc430a04 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:06:57.554226 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc56558-82b9-42b2-b79e-a132fc430a04 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:06:57.554563 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc56558-82b9-42b2-b79e-a132fc430a04 admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:06:57.555205 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adc56558-82b9-42b2-b79e-a132fc430a04 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:06:57.556846 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adc56558-82b9-42b2-b79e-a132fc430a04 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:06:57.566695 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1458/2884] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 01:06:57 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 01:06:57.571181 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec277914-8655-4ff9-ac75-7f51b8048b23 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:06:57.580737 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec277914-8655-4ff9-ac75-7f51b8048b23 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:06:57.585375 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8c75bc4-b8a3-4321-bca9-6ec5bdbf696b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:06:57.586528 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8c75bc4-b8a3-4321-bca9-6ec5bdbf696b admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:06:57.586939 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8c75bc4-b8a3-4321-bca9-6ec5bdbf696b admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:06:57.587309 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8c75bc4-b8a3-4321-bca9-6ec5bdbf696b admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:06:57.588025 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8c75bc4-b8a3-4321-bca9-6ec5bdbf696b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:06:57.589584 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8c75bc4-b8a3-4321-bca9-6ec5bdbf696b admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:06:57.600094 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1427/2885] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 01:06:57 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 01:06:57.604260 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adc56558-82b9-42b2-b79e-a132fc430a04 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:06:57.614517 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adc56558-82b9-42b2-b79e-a132fc430a04 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:06:57.619819 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c776b0-3a85-4563-b273-edde89f74d2d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:06:57.620457 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c776b0-3a85-4563-b273-edde89f74d2d admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:06:57.620737 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c776b0-3a85-4563-b273-edde89f74d2d admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:06:57.621012 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c776b0-3a85-4563-b273-edde89f74d2d admin admin] PATH_INFO: `/v3/projects/eca3b6a4954f446c872e43cb98c99528/users/f2791fd8579d4906bb7e341c6487f392/roles/588016933bab436d8b7c3ac76827356d` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:06:57.636066 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0c776b0-3a85-4563-b273-edde89f74d2d admin admin] RBAC: Authorizing `identity:create_grant(project_id=eca3b6a4954f446c872e43cb98c99528, user_id=f2791fd8579d4906bb7e341c6487f392, role_id=588016933bab436d8b7c3ac76827356d)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:06:57.638697 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0c776b0-3a85-4563-b273-edde89f74d2d admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:06:57.651144 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1459/2886] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 01:06:57 2022] PUT /identity/v3/projects/eca3b6a4954f446c872e43cb98c99528/users/f2791fd8579d4906bb7e341c6487f392/roles/588016933bab436d8b7c3ac76827356d => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:06:57.656413 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae28e4c0-6751-4b61-8559-959152358ee0 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:06:57.656697 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae28e4c0-6751-4b61-8559-959152358ee0 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:06:57.657064 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae28e4c0-6751-4b61-8559-959152358ee0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:06:57.675500 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-ae28e4c0-6751-4b61-8559-959152358ee0 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:06:57.676721 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-ae28e4c0-6751-4b61-8559-959152358ee0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:06:57.733794 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1428/2887] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:06:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:06:57.743878 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c776b0-3a85-4563-b273-edde89f74d2d admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:06:57.745557 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-a0c776b0-3a85-4563-b273-edde89f74d2d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:06:57.804536 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c776b0-3a85-4563-b273-edde89f74d2d admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:06:57.808130 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48ba0630-0220-455b-abc0-155abec7877c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:06:57.808728 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48ba0630-0220-455b-abc0-155abec7877c service neutron] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:06:57.809003 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48ba0630-0220-455b-abc0-155abec7877c service neutron] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:06:57.809274 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48ba0630-0220-455b-abc0-155abec7877c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:06:57.818035 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48ba0630-0220-455b-abc0-155abec7877c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:06:57.819453 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48ba0630-0220-455b-abc0-155abec7877c service neutron] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:06:57.828803 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1460/2888] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 01:06:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 01:06:58.446339 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac154895-7de6-4b1e-8c32-4e5c19e08ea8 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:06:58.446888 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac154895-7de6-4b1e-8c32-4e5c19e08ea8 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:06:58.446888 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac154895-7de6-4b1e-8c32-4e5c19e08ea8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:06:58.475141 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-ac154895-7de6-4b1e-8c32-4e5c19e08ea8 None None] MFA Rules not processed for user `f2791fd8579d4906bb7e341c6487f392`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:06:58.476436 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-ac154895-7de6-4b1e-8c32-4e5c19e08ea8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:06:58.525271 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:06:58.544114 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1429/2889] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:06:58 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:06:58.555435 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48ba0630-0220-455b-abc0-155abec7877c service neutron] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:06:58.556976 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-48ba0630-0220-455b-abc0-155abec7877c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:06:58.614944 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48ba0630-0220-455b-abc0-155abec7877c service neutron] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:06:58.619454 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5774a5df-4013-45c3-aa54-319498329440 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f3f48335668a4e8f934ac11ff66cd308', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:06:58.620091 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5774a5df-4013-45c3-aa54-319498329440 service cinder] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:06:58.620380 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5774a5df-4013-45c3-aa54-319498329440 service cinder] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:06:58.620655 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5774a5df-4013-45c3-aa54-319498329440 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:06:58.629129 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5774a5df-4013-45c3-aa54-319498329440 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:06:58.630555 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5774a5df-4013-45c3-aa54-319498329440 service cinder] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:06:58.640811 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1461/2890] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 01:06:58 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 01:06:59.999835 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6794bed-71dd-4422-8309-7755ef8ca19d None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:07:00.000374 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6794bed-71dd-4422-8309-7755ef8ca19d None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:07:00.000374 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6794bed-71dd-4422-8309-7755ef8ca19d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:07:00.018990 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-c6794bed-71dd-4422-8309-7755ef8ca19d None None] MFA Rules not processed for user `f2791fd8579d4906bb7e341c6487f392`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:07:00.020218 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-c6794bed-71dd-4422-8309-7755ef8ca19d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:07:00.027405 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1430/2891] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:06:59 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:07:00.565859 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5774a5df-4013-45c3-aa54-319498329440 service cinder] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:07:00.567381 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-5774a5df-4013-45c3-aa54-319498329440 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:07:00.638908 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5774a5df-4013-45c3-aa54-319498329440 service cinder] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:07:00.643874 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdcb506e-c2d4-4a34-a488-8af672cd28d1 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd25e240cb93643ed8122ccac9586936f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:07:00.644471 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdcb506e-c2d4-4a34-a488-8af672cd28d1 service placement] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:07:00.644748 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdcb506e-c2d4-4a34-a488-8af672cd28d1 service placement] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:07:00.645020 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdcb506e-c2d4-4a34-a488-8af672cd28d1 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:07:00.646730 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-cdcb506e-c2d4-4a34-a488-8af672cd28d1 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:07:00.658155 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdcb506e-c2d4-4a34-a488-8af672cd28d1 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:07:00.659754 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdcb506e-c2d4-4a34-a488-8af672cd28d1 service placement] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:07:00.669488 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1462/2892] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 01:07:00 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 01:07:02.398495 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6794bed-71dd-4422-8309-7755ef8ca19d None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:07:02.408692 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6794bed-71dd-4422-8309-7755ef8ca19d None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:07:02.412244 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51c06546-1c28-4ad6-a392-9cf78c51527d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:07:02.412834 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c06546-1c28-4ad6-a392-9cf78c51527d service neutron] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:07:02.413113 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c06546-1c28-4ad6-a392-9cf78c51527d service neutron] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:07:02.413380 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c06546-1c28-4ad6-a392-9cf78c51527d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:07:02.415684 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-51c06546-1c28-4ad6-a392-9cf78c51527d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:07:02.424924 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51c06546-1c28-4ad6-a392-9cf78c51527d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:07:02.426320 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51c06546-1c28-4ad6-a392-9cf78c51527d service neutron] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:07:02.434744 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1431/2893] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 01:07:02 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 01:07:09.621027 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdcb506e-c2d4-4a34-a488-8af672cd28d1 service placement] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:07:09.622508 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-cdcb506e-c2d4-4a34-a488-8af672cd28d1 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:07:09.627736 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:07:09.698559 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdcb506e-c2d4-4a34-a488-8af672cd28d1 service placement] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:07:09.704023 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f81d4e6-9481-4cbf-a64c-4f061025c0be service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a5d95969e74157bb2d2a57207e1651', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:07:09.704616 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f81d4e6-9481-4cbf-a64c-4f061025c0be service nova] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:07:09.704892 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f81d4e6-9481-4cbf-a64c-4f061025c0be service nova] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:07:09.705162 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f81d4e6-9481-4cbf-a64c-4f061025c0be service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:07:09.706844 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-6f81d4e6-9481-4cbf-a64c-4f061025c0be service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:07:09.718591 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f81d4e6-9481-4cbf-a64c-4f061025c0be service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:07:09.720194 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f81d4e6-9481-4cbf-a64c-4f061025c0be service nova] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:07:09.730501 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1463/2894] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 01:07:09 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 110 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 01:07:12.430490 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=13 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:07:19.726118 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=33 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:08:00.026404 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d207e55-2ff1-4135-ae54-679219772858 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:08:00.026932 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d207e55-2ff1-4135-ae54-679219772858 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:08:00.026932 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d207e55-2ff1-4135-ae54-679219772858 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:08:00.046635 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-7d207e55-2ff1-4135-ae54-679219772858 None None] MFA Rules not processed for user `f2791fd8579d4906bb7e341c6487f392`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:08:00.047844 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-7d207e55-2ff1-4135-ae54-679219772858 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:08:00.054481 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1432/2895] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:08:00 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:08:10.041218 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=3 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:08:54.505745 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f81d4e6-9481-4cbf-a64c-4f061025c0be service nova] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:08:54.517965 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f81d4e6-9481-4cbf-a64c-4f061025c0be service nova] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:08:54.522458 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22a4240b-2507-4918-baf0-24c09141941e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:08:54.523113 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a4240b-2507-4918-baf0-24c09141941e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:08:54.523375 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a4240b-2507-4918-baf0-24c09141941e admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:08:54.523673 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a4240b-2507-4918-baf0-24c09141941e admin admin] PATH_INFO: `/v3/users/f2791fd8579d4906bb7e341c6487f392` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:08:54.524772 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22a4240b-2507-4918-baf0-24c09141941e admin admin] RBAC: Authorizing `identity:delete_user(user_id=f2791fd8579d4906bb7e341c6487f392)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:08:54.526677 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22a4240b-2507-4918-baf0-24c09141941e admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:08:54.572456 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-22a4240b-2507-4918-baf0-24c09141941e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f2791fd8579d4906bb7e341c6487f392'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:08:54.579211 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-22a4240b-2507-4918-baf0-24c09141941e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f2791fd8579d4906bb7e341c6487f392'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:08:54.579884 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-22a4240b-2507-4918-baf0-24c09141941e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f2791fd8579d4906bb7e341c6487f392'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:08:54.580485 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-22a4240b-2507-4918-baf0-24c09141941e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f2791fd8579d4906bb7e341c6487f392'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:08:54.587552 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-22a4240b-2507-4918-baf0-24c09141941e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f2791fd8579d4906bb7e341c6487f392'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:08:54.606730 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1464/2896] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 01:08:54 2022] DELETE /identity/v3/users/f2791fd8579d4906bb7e341c6487f392 => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:08:54.779639 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d207e55-2ff1-4135-ae54-679219772858 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:08:54.792753 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d207e55-2ff1-4135-ae54-679219772858 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:08:54.851227 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce3d9286-3a76-46f4-a465-02b564a8fee4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:08:54.851939 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce3d9286-3a76-46f4-a465-02b564a8fee4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:08:54.852216 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce3d9286-3a76-46f4-a465-02b564a8fee4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:08:54.852483 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce3d9286-3a76-46f4-a465-02b564a8fee4 admin admin] PATH_INFO: `/v3/projects/eca3b6a4954f446c872e43cb98c99528` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:08:54.853598 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce3d9286-3a76-46f4-a465-02b564a8fee4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=eca3b6a4954f446c872e43cb98c99528)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:08:54.855469 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce3d9286-3a76-46f4-a465-02b564a8fee4 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:08:54.900582 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-ce3d9286-3a76-46f4-a465-02b564a8fee4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'eca3b6a4954f446c872e43cb98c99528', 'request_id': 'req-ce3d9286-3a76-46f4-a465-02b564a8fee4'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:08:54.906438 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-ce3d9286-3a76-46f4-a465-02b564a8fee4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'eca3b6a4954f446c872e43cb98c99528', 'request_id': 'req-ce3d9286-3a76-46f4-a465-02b564a8fee4'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:08:54.912422 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-ce3d9286-3a76-46f4-a465-02b564a8fee4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'eca3b6a4954f446c872e43cb98c99528', 'request_id': 'req-ce3d9286-3a76-46f4-a465-02b564a8fee4'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:08:54.925430 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-ce3d9286-3a76-46f4-a465-02b564a8fee4 admin admin] The token cache is being invalidate because project eca3b6a4954f446c872e43cb98c99528 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81453) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 01:08:54.925721 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-ce3d9286-3a76-46f4-a465-02b564a8fee4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:08:54.926347 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-ce3d9286-3a76-46f4-a465-02b564a8fee4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:08:54.927894 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1433/2897] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 01:08:54 2022] DELETE /identity/v3/projects/eca3b6a4954f446c872e43cb98c99528 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:08:54.932820 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15efb06e-2f73-4748-a38f-023267154898 None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:08:54.933108 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15efb06e-2f73-4748-a38f-023267154898 None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:08:54.933382 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15efb06e-2f73-4748-a38f-023267154898 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:08:54.953446 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-15efb06e-2f73-4748-a38f-023267154898 None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:08:54.954629 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-15efb06e-2f73-4748-a38f-023267154898 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:08:55.012297 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1465/2898] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:08:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:08:55.020811 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2929379a-3716-410c-a700-23cfcd711a1c None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:08:55.021101 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2929379a-3716-410c-a700-23cfcd711a1c None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:08:55.021381 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2929379a-3716-410c-a700-23cfcd711a1c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:08:55.041013 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-2929379a-3716-410c-a700-23cfcd711a1c None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:08:55.042247 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-2929379a-3716-410c-a700-23cfcd711a1c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:08:55.050815 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1434/2899] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:08:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:08:55.054882 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15efb06e-2f73-4748-a38f-023267154898 None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:08:55.064883 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15efb06e-2f73-4748-a38f-023267154898 None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:08:55.069321 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41dfce60-60ff-473e-9a5f-34b2e8d25680 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:08:55.069903 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41dfce60-60ff-473e-9a5f-34b2e8d25680 admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:08:55.070178 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41dfce60-60ff-473e-9a5f-34b2e8d25680 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:08:55.070448 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41dfce60-60ff-473e-9a5f-34b2e8d25680 admin admin] PATH_INFO: `/v3/domains` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:08:55.071081 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41dfce60-60ff-473e-9a5f-34b2e8d25680 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 09 01:08:55.071374 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41dfce60-60ff-473e-9a5f-34b2e8d25680 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:08:55.073021 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41dfce60-60ff-473e-9a5f-34b2e8d25680 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:08:55.084722 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1466/2900] 10.222.0.33 () {58 vars in 1153 bytes} [Sun Oct 9 01:08:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 09 01:08:55.089068 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2929379a-3716-410c-a700-23cfcd711a1c None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:08:55.098206 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2929379a-3716-410c-a700-23cfcd711a1c None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:08:55.102897 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d970274c-ac58-42a1-a3bf-b2b1e85ef67a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:08:55.103486 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d970274c-ac58-42a1-a3bf-b2b1e85ef67a admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:08:55.103804 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d970274c-ac58-42a1-a3bf-b2b1e85ef67a admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:08:55.104079 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d970274c-ac58-42a1-a3bf-b2b1e85ef67a admin admin] PATH_INFO: `/v3/projects` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:08:55.104697 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d970274c-ac58-42a1-a3bf-b2b1e85ef67a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:08:55.106569 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d970274c-ac58-42a1-a3bf-b2b1e85ef67a admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:08:55.133174 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1435/2901] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 01:08:55 2022] POST /identity/v3/projects => generated 371 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 01:08:55.137266 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41dfce60-60ff-473e-9a5f-34b2e8d25680 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:08:55.146985 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41dfce60-60ff-473e-9a5f-34b2e8d25680 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:08:55.201245 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e014ff79-970e-40b5-8f26-07f2a987ae00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:08:55.201876 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e014ff79-970e-40b5-8f26-07f2a987ae00 admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:08:55.202106 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e014ff79-970e-40b5-8f26-07f2a987ae00 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:08:55.202374 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e014ff79-970e-40b5-8f26-07f2a987ae00 admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:08:55.203128 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e014ff79-970e-40b5-8f26-07f2a987ae00 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:08:55.205060 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e014ff79-970e-40b5-8f26-07f2a987ae00 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:08:55.212841 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:08:55.228306 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1467/2902] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 01:08:55 2022] POST /identity/v3/users => generated 286 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 01:08:55.232934 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d970274c-ac58-42a1-a3bf-b2b1e85ef67a admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:08:55.242848 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d970274c-ac58-42a1-a3bf-b2b1e85ef67a admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:08:55.247372 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d642fc5-4687-49c6-a06b-bd247df12025 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:08:55.248002 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d642fc5-4687-49c6-a06b-bd247df12025 admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:08:55.248281 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d642fc5-4687-49c6-a06b-bd247df12025 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:08:55.248555 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d642fc5-4687-49c6-a06b-bd247df12025 admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:08:55.249179 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d642fc5-4687-49c6-a06b-bd247df12025 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:08:55.250908 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d642fc5-4687-49c6-a06b-bd247df12025 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:08:55.260609 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1436/2903] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 01:08:55 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 01:08:55.264198 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e014ff79-970e-40b5-8f26-07f2a987ae00 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:08:55.273972 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e014ff79-970e-40b5-8f26-07f2a987ae00 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:08:55.278576 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c5f87d7-852e-43dd-ac7e-3a466eb09b9f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:08:55.279683 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c5f87d7-852e-43dd-ac7e-3a466eb09b9f admin admin] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:08:55.279965 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c5f87d7-852e-43dd-ac7e-3a466eb09b9f admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:08:55.280236 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c5f87d7-852e-43dd-ac7e-3a466eb09b9f admin admin] PATH_INFO: `/v3/roles` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:08:55.280866 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c5f87d7-852e-43dd-ac7e-3a466eb09b9f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:08:55.282701 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c5f87d7-852e-43dd-ac7e-3a466eb09b9f admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:08:55.293116 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1468/2904] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 01:08:55 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 01:08:55.297664 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d642fc5-4687-49c6-a06b-bd247df12025 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:08:55.307233 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d642fc5-4687-49c6-a06b-bd247df12025 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:08:55.311806 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-743c57da-133b-40f8-be0d-706d5ce788f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:08:55.312438 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-743c57da-133b-40f8-be0d-706d5ce788f5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:08:55.312722 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-743c57da-133b-40f8-be0d-706d5ce788f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:08:55.312999 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-743c57da-133b-40f8-be0d-706d5ce788f5 admin admin] PATH_INFO: `/v3/projects/1fef9408db82475ab8903f48e483dd1b/users/72bdabb024674f48927b902d2225e473/roles/588016933bab436d8b7c3ac76827356d` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:08:55.328767 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-743c57da-133b-40f8-be0d-706d5ce788f5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1fef9408db82475ab8903f48e483dd1b, user_id=72bdabb024674f48927b902d2225e473, role_id=588016933bab436d8b7c3ac76827356d)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:08:55.331383 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-743c57da-133b-40f8-be0d-706d5ce788f5 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:08:55.345043 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1437/2905] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 01:08:55 2022] PUT /identity/v3/projects/1fef9408db82475ab8903f48e483dd1b/users/72bdabb024674f48927b902d2225e473/roles/588016933bab436d8b7c3ac76827356d => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:08:55.350284 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc2a093-4113-4477-88c0-6134b39bfe8c None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:08:55.350598 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc2a093-4113-4477-88c0-6134b39bfe8c None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:08:55.350986 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc2a093-4113-4477-88c0-6134b39bfe8c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:08:55.370189 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-6bc2a093-4113-4477-88c0-6134b39bfe8c None None] MFA Rules not processed for user `d9c8218a78034540ba9ed09b697588a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:08:55.371399 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-6bc2a093-4113-4477-88c0-6134b39bfe8c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:08:55.430560 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1469/2906] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:08:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:08:55.440110 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-743c57da-133b-40f8-be0d-706d5ce788f5 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:08:55.441773 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-743c57da-133b-40f8-be0d-706d5ce788f5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:08:55.471853 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:08:55.499653 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-743c57da-133b-40f8-be0d-706d5ce788f5 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:08:55.503225 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-539dc783-71b7-4b99-a23e-e31459f266f5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7d2eabf9865847ceb6e44d55126078a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:08:55.503857 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-539dc783-71b7-4b99-a23e-e31459f266f5 service neutron] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:08:55.504138 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-539dc783-71b7-4b99-a23e-e31459f266f5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:08:55.504428 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-539dc783-71b7-4b99-a23e-e31459f266f5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:08:55.513185 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-539dc783-71b7-4b99-a23e-e31459f266f5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:08:55.514894 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-539dc783-71b7-4b99-a23e-e31459f266f5 service neutron] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:08:55.524061 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1438/2907] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 01:08:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 09 01:08:59.549771 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85e7787-7ea4-4ef0-a75e-ebef9ae3d23a None None] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:08:59.550335 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85e7787-7ea4-4ef0-a75e-ebef9ae3d23a None None] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:08:59.550335 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85e7787-7ea4-4ef0-a75e-ebef9ae3d23a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:08:59.598185 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.auth.core [None req-a85e7787-7ea4-4ef0-a75e-ebef9ae3d23a None None] MFA Rules not processed for user `72bdabb024674f48927b902d2225e473`. Rule list: `[]` (Enabled: `True`). {{(pid=81452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:08:59.599673 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-a85e7787-7ea4-4ef0-a75e-ebef9ae3d23a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:08:59.679416 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1470/2908] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:08:59 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:08:59.684055 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-539dc783-71b7-4b99-a23e-e31459f266f5 service neutron] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:08:59.695996 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-539dc783-71b7-4b99-a23e-e31459f266f5 service neutron] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:08:59.703767 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c74184d-3468-45f5-8611-54947836ba91 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:08:59.704390 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c74184d-3468-45f5-8611-54947836ba91 admin admin] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:08:59.704698 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c74184d-3468-45f5-8611-54947836ba91 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:08:59.704973 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c74184d-3468-45f5-8611-54947836ba91 admin admin] PATH_INFO: `/v3/projects` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:08:59.705610 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c74184d-3468-45f5-8611-54947836ba91 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:08:59.708512 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c74184d-3468-45f5-8611-54947836ba91 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:08:59.747263 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1439/2909] 10.222.0.33 () {60 vars in 1153 bytes} [Sun Oct 9 01:08:59 2022] POST /identity/v3/projects => generated 373 bytes in 64 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 09 01:08:59.748863 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a85e7787-7ea4-4ef0-a75e-ebef9ae3d23a None None] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:08:59.760502 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a85e7787-7ea4-4ef0-a75e-ebef9ae3d23a None None] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:08:59.823253 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e6b14fa-9890-4bd0-bfd7-f84b13e4b5df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:08:59.823816 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6b14fa-9890-4bd0-bfd7-f84b13e4b5df admin admin] REQUEST_METHOD: `POST` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:08:59.824096 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6b14fa-9890-4bd0-bfd7-f84b13e4b5df admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:08:59.824543 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6b14fa-9890-4bd0-bfd7-f84b13e4b5df admin admin] PATH_INFO: `/v3/users` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:08:59.825220 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e6b14fa-9890-4bd0-bfd7-f84b13e4b5df admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:08:59.827772 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e6b14fa-9890-4bd0-bfd7-f84b13e4b5df admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:08:59.844372 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:INSERT=6,keystone:SELECT=94 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:08:59.852080 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1471/2910] 10.222.0.33 () {60 vars in 1144 bytes} [Sun Oct 9 01:08:59 2022] POST /identity/v3/users => generated 293 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 09 01:08:59.855387 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c74184d-3468-45f5-8611-54947836ba91 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:08:59.866618 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c74184d-3468-45f5-8611-54947836ba91 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:08:59.871717 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b8d9a8c-1f94-4e19-b5ad-2461734aa879 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:08:59.872346 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b8d9a8c-1f94-4e19-b5ad-2461734aa879 admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:08:59.873281 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b8d9a8c-1f94-4e19-b5ad-2461734aa879 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:08:59.873559 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b8d9a8c-1f94-4e19-b5ad-2461734aa879 admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:08:59.874209 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b8d9a8c-1f94-4e19-b5ad-2461734aa879 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:08:59.875908 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b8d9a8c-1f94-4e19-b5ad-2461734aa879 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:08:59.887366 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1440/2911] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 01:08:59 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 09 01:08:59.892316 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e6b14fa-9890-4bd0-bfd7-f84b13e4b5df admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:08:59.902901 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e6b14fa-9890-4bd0-bfd7-f84b13e4b5df admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:08:59.908012 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2987c41-5bbb-4950-973e-ee54deb9e9e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:08:59.908670 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2987c41-5bbb-4950-973e-ee54deb9e9e8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:08:59.908948 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2987c41-5bbb-4950-973e-ee54deb9e9e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:08:59.909242 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2987c41-5bbb-4950-973e-ee54deb9e9e8 admin admin] PATH_INFO: `/v3/projects/ab20088cee27434ab33024d0693c44bd/users/618462aad0144d86b86dad54e0a3996a/roles/092036ee46ac427baf4d56468575f697` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:08:59.927262 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2987c41-5bbb-4950-973e-ee54deb9e9e8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ab20088cee27434ab33024d0693c44bd, user_id=618462aad0144d86b86dad54e0a3996a, role_id=092036ee46ac427baf4d56468575f697)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:08:59.930179 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2987c41-5bbb-4950-973e-ee54deb9e9e8 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:08:59.948779 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1472/2912] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 01:08:59 2022] PUT /identity/v3/projects/ab20088cee27434ab33024d0693c44bd/users/618462aad0144d86b86dad54e0a3996a/roles/092036ee46ac427baf4d56468575f697 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 09 01:08:59.953316 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b8d9a8c-1f94-4e19-b5ad-2461734aa879 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:08:59.971707 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b8d9a8c-1f94-4e19-b5ad-2461734aa879 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:09:00.032170 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d2996df-fddc-4147-92ee-6be03280b733 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:09:00.032781 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d2996df-fddc-4147-92ee-6be03280b733 admin admin] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:09:00.033186 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d2996df-fddc-4147-92ee-6be03280b733 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:09:00.033459 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d2996df-fddc-4147-92ee-6be03280b733 admin admin] PATH_INFO: `/v3/roles` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:09:00.034090 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d2996df-fddc-4147-92ee-6be03280b733 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:09:00.035705 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d2996df-fddc-4147-92ee-6be03280b733 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:09:00.046341 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1441/2913] 10.222.0.33 () {58 vars in 1122 bytes} [Sun Oct 9 01:08:59 2022] GET /identity/v3/roles => generated 1420 bytes in 94 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 09 01:09:00.050827 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2987c41-5bbb-4950-973e-ee54deb9e9e8 admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:09:00.061970 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2987c41-5bbb-4950-973e-ee54deb9e9e8 admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:09:00.066632 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12ae6274-181b-4d59-96d7-1922e6b4bb6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:09:00.067281 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ae6274-181b-4d59-96d7-1922e6b4bb6c admin admin] REQUEST_METHOD: `PUT` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:09:00.067561 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ae6274-181b-4d59-96d7-1922e6b4bb6c admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:09:00.067892 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ae6274-181b-4d59-96d7-1922e6b4bb6c admin admin] PATH_INFO: `/v3/projects/ab20088cee27434ab33024d0693c44bd/users/618462aad0144d86b86dad54e0a3996a/roles/092036ee46ac427baf4d56468575f697` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:09:00.069791 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12ae6274-181b-4d59-96d7-1922e6b4bb6c admin admin] RBAC: Authorizing `identity:create_grant(project_id=ab20088cee27434ab33024d0693c44bd, user_id=618462aad0144d86b86dad54e0a3996a, role_id=092036ee46ac427baf4d56468575f697)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:09:00.072545 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12ae6274-181b-4d59-96d7-1922e6b4bb6c admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:09:00.084633 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1473/2914] 10.222.0.33 () {60 vars in 1483 bytes} [Sun Oct 9 01:09:00 2022] PUT /identity/v3/projects/ab20088cee27434ab33024d0693c44bd/users/618462aad0144d86b86dad54e0a3996a/roles/092036ee46ac427baf4d56468575f697 => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:09:04.418489 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd6b31d-a106-46f5-8e3a-bfd63143078b None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:09:04.418989 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd6b31d-a106-46f5-8e3a-bfd63143078b None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:09:04.419118 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd6b31d-a106-46f5-8e3a-bfd63143078b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:09:04.454274 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-edd6b31d-a106-46f5-8e3a-bfd63143078b None None] MFA Rules not processed for user `618462aad0144d86b86dad54e0a3996a`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:09:04.455506 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-edd6b31d-a106-46f5-8e3a-bfd63143078b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:09:04.535908 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1442/2915] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:09:04 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:09:04.551481 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12ae6274-181b-4d59-96d7-1922e6b4bb6c admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:09:04.553493 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-12ae6274-181b-4d59-96d7-1922e6b4bb6c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:09:04.632086 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12ae6274-181b-4d59-96d7-1922e6b4bb6c admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:09:04.637642 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32f674ff-d284-4714-85e8-425e4b0651e5 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e0a5d95969e74157bb2d2a57207e1651', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['member', 'ResellerAdmin', 'admin', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:09:04.638784 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f674ff-d284-4714-85e8-425e4b0651e5 service nova] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:09:04.639067 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f674ff-d284-4714-85e8-425e4b0651e5 service nova] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:09:04.639341 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f674ff-d284-4714-85e8-425e4b0651e5 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:09:04.696752 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32f674ff-d284-4714-85e8-425e4b0651e5 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:09:04.698356 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32f674ff-d284-4714-85e8-425e4b0651e5 service nova] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:09:04.708303 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1474/2916] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 01:09:04 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 157 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 01:09:06.749960 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d9ef416-7bb2-4cdd-9db7-84bc8e8e76c2 None None] REQUEST_METHOD: `POST` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:09:06.750442 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d9ef416-7bb2-4cdd-9db7-84bc8e8e76c2 None None] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:09:06.750572 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d9ef416-7bb2-4cdd-9db7-84bc8e8e76c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:09:06.770320 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.auth.core [None req-3d9ef416-7bb2-4cdd-9db7-84bc8e8e76c2 None None] MFA Rules not processed for user `72bdabb024674f48927b902d2225e473`. Rule list: `[]` (Enabled: `True`). {{(pid=81453) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 09 01:09:06.771561 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-3d9ef416-7bb2-4cdd-9db7-84bc8e8e76c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:09:06.778502 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1443/2917] 10.222.0.33 () {58 vars in 958 bytes} [Sun Oct 9 01:09:06 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 09 01:09:14.703966 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=72,keystone:INSERT=2 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:09:16.764461 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=94,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:10:00.388373 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32f674ff-d284-4714-85e8-425e4b0651e5 service nova] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:10:00.390449 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-32f674ff-d284-4714-85e8-425e4b0651e5 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:10:00.457531 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32f674ff-d284-4714-85e8-425e4b0651e5 service nova] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:10:00.462442 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f1712a0-28ae-466f-8602-5c1d5bfde74d service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd25e240cb93643ed8122ccac9586936f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:10:00.463033 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1712a0-28ae-466f-8602-5c1d5bfde74d service placement] REQUEST_METHOD: `GET` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:10:00.463311 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1712a0-28ae-466f-8602-5c1d5bfde74d service placement] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:10:00.463584 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1712a0-28ae-466f-8602-5c1d5bfde74d service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:10:00.465296 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-9f1712a0-28ae-466f-8602-5c1d5bfde74d service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:10:00.476423 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f1712a0-28ae-466f-8602-5c1d5bfde74d service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:10:00.477987 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f1712a0-28ae-466f-8602-5c1d5bfde74d service placement] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:10:00.487757 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1475/2918] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 01:10:00 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 01:10:10.483444 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=32 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:10:27.422099 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d9ef416-7bb2-4cdd-9db7-84bc8e8e76c2 None None] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:10:27.496684 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d9ef416-7bb2-4cdd-9db7-84bc8e8e76c2 None None] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:10:27.506200 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4223a2b7-fa4a-4630-9195-29bf8ff9d5fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:10:27.507091 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4223a2b7-fa4a-4630-9195-29bf8ff9d5fc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:10:27.507474 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4223a2b7-fa4a-4630-9195-29bf8ff9d5fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:10:27.507913 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4223a2b7-fa4a-4630-9195-29bf8ff9d5fc admin admin] PATH_INFO: `/v3/users/72bdabb024674f48927b902d2225e473` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:10:27.509230 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4223a2b7-fa4a-4630-9195-29bf8ff9d5fc admin admin] RBAC: Authorizing `identity:delete_user(user_id=72bdabb024674f48927b902d2225e473)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:10:27.511329 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4223a2b7-fa4a-4630-9195-29bf8ff9d5fc admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:10:27.576445 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-4223a2b7-fa4a-4630-9195-29bf8ff9d5fc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '72bdabb024674f48927b902d2225e473'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:10:27.585377 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-4223a2b7-fa4a-4630-9195-29bf8ff9d5fc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '72bdabb024674f48927b902d2225e473'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:10:27.586080 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-4223a2b7-fa4a-4630-9195-29bf8ff9d5fc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '72bdabb024674f48927b902d2225e473'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:10:27.593185 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-4223a2b7-fa4a-4630-9195-29bf8ff9d5fc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '72bdabb024674f48927b902d2225e473'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:10:27.606034 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-4223a2b7-fa4a-4630-9195-29bf8ff9d5fc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '72bdabb024674f48927b902d2225e473'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:10:27.612548 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1444/2919] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 01:10:27 2022] DELETE /identity/v3/users/72bdabb024674f48927b902d2225e473 => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:10:27.784371 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f1712a0-28ae-466f-8602-5c1d5bfde74d service placement] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:10:27.796498 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f1712a0-28ae-466f-8602-5c1d5bfde74d service placement] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:10:27.852765 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e52ccba9-d8b6-41c5-908d-a66f48a7dfbb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:10:27.853384 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e52ccba9-d8b6-41c5-908d-a66f48a7dfbb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:10:27.853662 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e52ccba9-d8b6-41c5-908d-a66f48a7dfbb admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:10:27.853934 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e52ccba9-d8b6-41c5-908d-a66f48a7dfbb admin admin] PATH_INFO: `/v3/projects/1fef9408db82475ab8903f48e483dd1b` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:10:27.855022 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e52ccba9-d8b6-41c5-908d-a66f48a7dfbb admin admin] RBAC: Authorizing `identity:delete_project(project_id=1fef9408db82475ab8903f48e483dd1b)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:10:27.856929 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e52ccba9-d8b6-41c5-908d-a66f48a7dfbb admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:10:27.906302 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-e52ccba9-d8b6-41c5-908d-a66f48a7dfbb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1fef9408db82475ab8903f48e483dd1b', 'request_id': 'req-e52ccba9-d8b6-41c5-908d-a66f48a7dfbb'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:10:27.912135 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-e52ccba9-d8b6-41c5-908d-a66f48a7dfbb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1fef9408db82475ab8903f48e483dd1b', 'request_id': 'req-e52ccba9-d8b6-41c5-908d-a66f48a7dfbb'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:10:27.919016 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-e52ccba9-d8b6-41c5-908d-a66f48a7dfbb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1fef9408db82475ab8903f48e483dd1b', 'request_id': 'req-e52ccba9-d8b6-41c5-908d-a66f48a7dfbb'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:10:27.934034 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-e52ccba9-d8b6-41c5-908d-a66f48a7dfbb admin admin] The token cache is being invalidate because project 1fef9408db82475ab8903f48e483dd1b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 01:10:27.934320 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-e52ccba9-d8b6-41c5-908d-a66f48a7dfbb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:10:27.935041 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-e52ccba9-d8b6-41c5-908d-a66f48a7dfbb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:10:27.936534 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1476/2920] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 01:10:27 2022] DELETE /identity/v3/projects/1fef9408db82475ab8903f48e483dd1b => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:10:27.941151 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4223a2b7-fa4a-4630-9195-29bf8ff9d5fc admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:10:27.942554 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-4223a2b7-fa4a-4630-9195-29bf8ff9d5fc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:10:27.955642 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4223a2b7-fa4a-4630-9195-29bf8ff9d5fc admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:10:28.011664 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb654708-5902-4155-866b-af90dcc4e885 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:10:28.012220 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb654708-5902-4155-866b-af90dcc4e885 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:10:28.012496 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb654708-5902-4155-866b-af90dcc4e885 admin admin] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:10:28.012767 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb654708-5902-4155-866b-af90dcc4e885 admin admin] PATH_INFO: `/v3/users/618462aad0144d86b86dad54e0a3996a` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:10:28.013877 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb654708-5902-4155-866b-af90dcc4e885 admin admin] RBAC: Authorizing `identity:delete_user(user_id=618462aad0144d86b86dad54e0a3996a)` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:10:28.015802 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb654708-5902-4155-866b-af90dcc4e885 admin admin] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:10:28.037739 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:10:28.063184 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-bb654708-5902-4155-866b-af90dcc4e885 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '618462aad0144d86b86dad54e0a3996a'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:10:28.070626 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-bb654708-5902-4155-866b-af90dcc4e885 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '618462aad0144d86b86dad54e0a3996a'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:10:28.071304 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-bb654708-5902-4155-866b-af90dcc4e885 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '618462aad0144d86b86dad54e0a3996a'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:10:28.085468 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-bb654708-5902-4155-866b-af90dcc4e885 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '618462aad0144d86b86dad54e0a3996a'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:10:28.099013 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.notifications [None req-bb654708-5902-4155-866b-af90dcc4e885 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '618462aad0144d86b86dad54e0a3996a'} {{(pid=81453) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:10:28.105409 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1445/2921] 10.222.0.33 () {58 vars in 1224 bytes} [Sun Oct 9 01:10:27 2022] DELETE /identity/v3/users/618462aad0144d86b86dad54e0a3996a => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:10:28.281638 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e52ccba9-d8b6-41c5-908d-a66f48a7dfbb admin admin] Authenticating user token {{(pid=81452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:10:28.283911 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.fernet_utils [None req-e52ccba9-d8b6-41c5-908d-a66f48a7dfbb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:10:28.299285 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e52ccba9-d8b6-41c5-908d-a66f48a7dfbb admin admin] Validating token access rules against request {{(pid=81452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:10:28.358833 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ad50bc1-f058-4252-a9e1-114e2e137645 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd9c8218a78034540ba9ed09b697588a8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '34e0fbee16724df3abcac7131a4822aa', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:10:28.359457 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ad50bc1-f058-4252-a9e1-114e2e137645 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:10:28.359780 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ad50bc1-f058-4252-a9e1-114e2e137645 admin admin] SCRIPT_NAME: `/identity` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:10:28.360115 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ad50bc1-f058-4252-a9e1-114e2e137645 admin admin] PATH_INFO: `/v3/projects/ab20088cee27434ab33024d0693c44bd` {{(pid=81452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:10:28.361774 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ad50bc1-f058-4252-a9e1-114e2e137645 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ab20088cee27434ab33024d0693c44bd)` {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:10:28.363713 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ad50bc1-f058-4252-a9e1-114e2e137645 admin admin] RBAC: Authorization granted {{(pid=81452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:10:28.397030 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:10:28.411485 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-5ad50bc1-f058-4252-a9e1-114e2e137645 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ab20088cee27434ab33024d0693c44bd', 'request_id': 'req-5ad50bc1-f058-4252-a9e1-114e2e137645'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:10:28.417017 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-5ad50bc1-f058-4252-a9e1-114e2e137645 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ab20088cee27434ab33024d0693c44bd', 'request_id': 'req-5ad50bc1-f058-4252-a9e1-114e2e137645'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:10:28.423502 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-5ad50bc1-f058-4252-a9e1-114e2e137645 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ab20088cee27434ab33024d0693c44bd', 'request_id': 'req-5ad50bc1-f058-4252-a9e1-114e2e137645'} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:10:28.436075 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-5ad50bc1-f058-4252-a9e1-114e2e137645 admin admin] The token cache is being invalidate because project ab20088cee27434ab33024d0693c44bd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 09 01:10:28.436368 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-5ad50bc1-f058-4252-a9e1-114e2e137645 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:10:28.437024 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG keystone.notifications [None req-5ad50bc1-f058-4252-a9e1-114e2e137645 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 09 01:10:28.438544 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: [pid: 81452|app: 0|req: 1477/2922] 10.222.0.33 () {58 vars in 1233 bytes} [Sun Oct 9 01:10:28 2022] DELETE /identity/v3/projects/ab20088cee27434ab33024d0693c44bd => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 09 01:10:38.101913 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:DELETE=7,keystone:SELECT=20,keystone:INSERT=1 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:10:38.432920 dv-cinder-master-3-iscsi devstack@keystone.service[81452]: DEBUG dbcounter [-] [81452] Writing DB stats keystone:SELECT=12,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 09 01:15:08.373583 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb654708-5902-4155-866b-af90dcc4e885 admin admin] Authenticating user token {{(pid=81453) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 09 01:15:08.374985 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-bb654708-5902-4155-866b-af90dcc4e885 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:15:08.513437 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb654708-5902-4155-866b-af90dcc4e885 admin admin] Validating token access rules against request {{(pid=81453) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 09 01:15:08.571737 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-200e0732-09b5-4476-b793-d78c76169ea1 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd25e240cb93643ed8122ccac9586936f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '013c83476b1340fbb051061833c4a9c6', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81453) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 09 01:15:08.572343 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-200e0732-09b5-4476-b793-d78c76169ea1 service placement] REQUEST_METHOD: `GET` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 09 01:15:08.572627 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-200e0732-09b5-4476-b793-d78c76169ea1 service placement] SCRIPT_NAME: `/identity` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 09 01:15:08.572901 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.server.flask.request_processing.req_logging [None req-200e0732-09b5-4476-b793-d78c76169ea1 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81453) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 09 01:15:08.574635 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.fernet_utils [None req-200e0732-09b5-4476-b793-d78c76169ea1 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81453) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 09 01:15:08.586223 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-200e0732-09b5-4476-b793-d78c76169ea1 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 09 01:15:08.587828 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-200e0732-09b5-4476-b793-d78c76169ea1 service placement] RBAC: Authorization granted {{(pid=81453) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 09 01:15:08.598176 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: [pid: 81453|app: 0|req: 1446/2923] 10.222.0.33 () {60 vars in 1365 bytes} [Sun Oct 9 01:15:08 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 225 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 09 01:15:18.593254 dv-cinder-master-3-iscsi devstack@keystone.service[81453]: DEBUG dbcounter [-] [81453] Writing DB stats keystone:SELECT=89 {{(pid=81453) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}